Brain Dynamics of Speech Modes Encoding: Loud and Whispered Speech Versus Standard Speech

Speakers can produce intelligible and accurate utterances almost automatically with a low error rate. Despite decades of investigation, a clear account of the interaction between the neural processes underlying speech production and their dynamics is still needed (Bohland et al. 2010; Laganaro 2019; Miller & Guenther 2021; Verwoert et al. 2022). Phonetic encoding [hereafter motor speech encoding processes] is the label given by some authors in the literature to describe the process of transforming an abstract linguistic sequence into a motor code readable by articulators (W. J. Levelt 1989; W. J. M. Levelt et al. 1999; Indefrey 2011; Guenther 2016). This encoding stage has been less studied in comparison to other language encoding processes resulting in a poor understanding of the underlying spatio-temporal dynamics (Indefrey 2011). The four level (FL) model from Van der Merwe (2020) proposed that motor speech encoding could be subdivided into two sequential substages; motor planning (i.e., retrieval of motor plans) and motor programming (i.e., where spatiotemporal and force dimensions are specified). This subdivision has been motivated by clinical symptoms of motor speech disorders such as apraxia of speech and dysarthria. To the best of our knowledge, the FL model is the only speech production model that provide some inputs regarding the encoding of speech modes. The model states that all the different speech modes can be grouped as a whole entity and would be thus encoded in the same way. Here, speech modes would be encoded through tuning of the unique suprasegmental features during the motor programming stage. Nevertheless, there is no empirical evidence corroborating this proposition. Studying the neural processes underlying speech modes as compared to standard speech can thus provide a relevant way of investigating whether speech modes require adjustments at specific encoding stages or whether they are supported by different encoding processes, as further presented below.

Speech modes constitute an omnipresent part of verbal exchanges. The way people speak is continuously influenced by intrinsic factors (speaker related) and extrinsic factors (environment or listener related) (Kelly & Hansen 2021; Smiljanić & Bradlow 2009; Whitfield et al. 2021). For instance, one needs to modulate his speech production to tell his friend the food he would like to order in a stadium full of supporters that are loudly singing an anthem. Moreover, meaningful cues (e.g., linguistic, affective or social cues) are conveyed to the interlocutor through the modulation of speech (Perkell 2012; Tourville & Guenther 2011). As an example, a speaker giving a talk during a conference will adopt a clear speech mode to emphasize his take home message. Zhang and Hansen (2007) proposed five speech modes with unique articulatory and phonatory features: whispered speech, soft speech, neutral (equivalent to standard) speech, loud speech and shouted speech. In this regard, it has been hypothesized that each speech mode involves its own mechanisms resulting in specific articulatory and phonatory patterns (Zhang & Hansen 2007; Scott, 2022). We will briefly describe the two speech modes that will be investigated in the present study, namely loud speech (LS) and whispered speech (WS).

When speakers struggle to convey a message to an interlocutor, for instance in a noisy environment, they usually modify their speech by increasing their vocal effort. In this case, increase in loudness leads to phonatory adjustments and changes in speech kinematics (Dromey & Ramig 1998; Huber & Chandrasekaran 2006; Whitfield et al. 2021). These manifestations associated to increased Sound Intensity Level (SIL) pertain to a specific speech mode labeled “LS”. Intuitively, one would conceptualize LS as the best speech mode to make yourself heard by someone else. However, the results obtained by Whitfield et al. (2021) indicate that LS’s main characteristic is increasing vocal intensity while there is not necessarily an improvement of the articulatory distinctiveness of the message conveyed. In the literature, LS has usually been associated to an increase in the standard SIL of 10 dB ± 4 dB (Huber & Chandrasekaran 2006; Whitfield et al. 2021). The encoding processes responsible for the increase in vocal loudness have not been clarified by functional neuroimaging or computational models, but some hypotheses on the proposed mechanisms will be presented below.

WS is a widespread mode of communication aiming at conveying a message while remaining discreet. This speech mode is convenient in situations requiring silence (e.g., movie, theatre) or to keep private the content of a message (e.g., telling a secret). The ability to whisper is specific to humans (Tsunoda et al. 2011) and is characterized by reduced intelligibility and perceptibility for the listener as well as a more effortful production from the speaker’s point of view (Zhang et al. 2018). During whispered speech, physiological adjustments are applied to specific muscles of the larynx in order to prevent vocal folds vibration (Konnai et al. 2017; Solomon et al. 1989; Tsunoda et al. 2011). This absence of phonation provides unique features to WS. Actually, among speech modes, phonetic features of WS characterize it as the most distinct speech mode in comparison to SS (Kelly & Hansen 2021; Zhang et al. 2018; Zhang & Hansen 2007). Similar to LS, no consensus has been reached in the literature regarding the encoding processes responsible for whispering, leading to several proposed hypotheses.

As anticipated previously, two possible hypotheses stem from the literature regarding the encoding processes associated to the production of loud and whispered utterances. On one hand, behavioral results (e.g., Huber & Chandrasekaran 2006) have led Whitfield et al., (2021) to hypothesize that an upregulation in the neuromotor drive is the mechanism at the origin of LS. However, it is unclear when and how this upregulation occurs in the motor speech encoding stage. On the other hand, two hypotheses were formulated based on functional Magnetic Resonance Imaging (fMRI) investigations in order to characterize the brain processes underlying WS. Correia et al. (2020) demonstrated that the fMRI response was greater for voiced speech than WS in the dorsal laryngeal motor area (dLMA), located in the primary motor cortex (M1). Under this hypothesis, the same brain mechanisms are at play for SS and WS, with larger recruitment for the former. A different hypothesis has been proposed by Tsunoda et al. (2011) based on a voluntary switching mechanism, in which ordinary speech would be transformed into whispering thanks to functional changes in the frontal lobe. However, their results showed two distinct patterns of brain activation in the frontal lobe involving both increased and decreased brain activation for WS relative to SS, which thus did not clarify how the functional switch would be carried out. In summary, the whispered speech’s literature gathers two distinct approaches concerning the encoding of WS: one described a functional difference in a specific motor region responsible for laryngeal control while another suggested the involvement of a voluntary functional switching mechanism during production of whispered utterances. In light of these theoretical propositions, speech modes could be encoded either (1) through neural adjustment of the same brain processes in the motor programming substage as proposed in the FL model or (2) through the involvement of an additional mechanism overlaying onto regular motor programming encoding processes. In particular, this study will explore these two hypotheses using behavioral and electrophysiological contrasts between speech modes and normally phonated speech. Specifically, LS (Experiment 1) and WS (Experiment 2) were compared to SS during a delayed production task of non-sense speech sequences (pseudowords). This paradigm is ideal to isolate motor speech encoding processes from linguistic encoding processes (Laganaro 2019, 2023; Piai et al. 2014). Electroencephalography (EEG)/Event-related potential (ERP) correlates of speech modes and SS will be analyzed during a time-window of about 350 ms preceding the vocal onset (hereafter referred to as “response-locked”) corresponding to the motor speech encoding stage. This time window is thus aligned to the vocal onset and analyzed in a backward fashion. In present study, we will exploit the high temporal resolution provided by the EEG to match the fast time scale of speech production processes (den Hollander et al. 2019; Laganaro & Perret 2011; Piai et al. 2014; Verwoert et al. 2022). Especially, we track the temporal dynamics of brain activations in the different experimental conditions via microstates analysis (Michel et al. 2009; Michel & Murray 2012; Murray et al. 2008) which will allow to investigate whether the encoding of speech modes elicit different brain processes relative to SS or if the same brain processes are engaged but with different dynamics.

The speech stimuli to be produced consisted of 67 monosyllabic and disyllabic pseudowords (see more details in Appendix A). Pseudowords were selected to avoid any linguistic effect related to words and thus focus on speech production. The pseudowords were composed of phonotactically legal French syllables according to the French database Lexique2 (New et al. 2004). All the items had the following syllabic structures: C₁C₂V₁ – C₃V₂ for the disyllabic items (e.g., trafa) and C₁C₂V₁ for the monosyllabic items (e.g., pra), with C₁ being one of the three following voiceless plosives: /p/, /t/ or /k/.

The experiment took place in a soundproof room in which participants were seated at about 70 cm from the computer screen. The software E-Prime 3.0 (Psychology Software Tools, Pittsburgh, PA) was used to present the stimuli in several experimental blocks and to record participants’ productions.

Participants performed a delayed production task (see Fig. 1), in which they were asked to prepare a speech sequence based on a written pseudoword and to produce it aloud when a cue (here a question mark) appeared on the screen. Each trial displayed in succession a fixation cross (350 ms), a pseudoword written in white at the center of a black screen (1200 ms), a sequence of three dots indicating a variable waiting delay (either 1300 or 1600 ms) and eventually a yellow question mark appeared on the screen (1700 ms). The question mark was the cue indicating to the participants to produce the pseudoword previously presented as quickly and as accurately as possible. In some cases, yellow ellipsis dots appeared on the screen instead of the question mark indicating that no production were expected. These “no-go” trials, although not analyzed, were integrated to keep participants’ attention and to avoid anticipatory responses using a varying delay of either 1000 or 1900 ms. On average, no-go items appeared approximatively every nine trials.

Before the beginning of the experiment, participants read aloud a list containing all the stimuli to ensure they pronounced them correctly. Halfway through, they were asked to produce the rest of the pseudowords by adopting a loud speech mode. In cases of incorrect pronunciation, they were first corrected and then asked to produce the pseudoword in its correct form. A short training session with five pseudowords produced normally (three go and two no-go trials) preceded the experiment to ensure that the participants were comfortable with the experimental procedure. The experiment was segmented in eight experimental blocks, including four blocks of standard speech (SS) and four blocks of loud Speech (LS), presented in an alternated manner (see Fig. 1 right panel). Each block contained between 48 and 50 stimuli with both pseudowords and no-go items. Across the eight blocks, participants produced the same 180 pseudowords in each condition. Before each SS block, participants were asked to speak as usual. Before each LS block, participants were instructed to speak louder than usual, aiming at being heard from outside the soundproof room. To ensure that participants produced utterances that were loud enough during loud blocks, intensity was checked by the experimenters on a sound level meter which was hidden from the participants. Half of the participants started the experiment with a block in SS (order 1) and the other half with LS (order 2). Short self-paced breaks were given to the participants between blocks. Four lists including the 360 pseudowords were created and were randomly assigned to the participants to avoid order effects. The speech productions were recorded for off-line accuracy (ACC) check and extraction of vocal onsets (or reaction times, RT).

Intensity was extracted and analyzed with the Praat software (Boersma & Van Heuven 2001). Speech intensity of all SS productions was averaged for each participant to establish an individual cut-off threshold. Therefore, loud utterances that were not higher than 8 dB in comparison to participant’s mean intensity were removed from the analyses. RT and ACC were extracted off-line through listening and visual inspection of the individual audio files with the Checkvocal Software (Protopapas 2007). Uncomplete (e.g., /kRat/ instead of /kRati/), uncertain (e.g.,/kr/…/kRata/) and incorrectly (e.g.,/kRotu/ instead of /kRutu/) produced pseudowords as well as productions that did not correspond to the target speech mode were considered as erroneous productions and were thus removed from the analyses. The vocal onset of each pseudoword was identified by aligning to the plosion bar produced by C₁. Two judges (i.e., first and third authors) listened to the entire speech dataset resulting in an inter-judge agreement of 98% for ACC and 91% for RT. As cleaning procedure, RT with a SD above 2.5 of the mean latency of production per participant and per condition were removed. As ACC was not part of our hypotheses, this metric will be used as a descriptive statistic. The behavioral results on RT were analyzed using the Mixed Model approach (Bates et al. 2014; Carson & Beeson 2013) with the R-Software (R Core Team 2021). We compared multiple nested models that were built up by adding one effect at the time. The best model (see Appendix B) contained RT as dependent variable; speech mode (SS and LS), order of the experimental blocks (loud first or standard first) and length (monosyllabic or disyllabic) as fixed effects and subjects and items as random variables. Interaction effects between speech mode and order of experimental blocks were also tested in the model.

The electrophysiological data was recorded continuously during the experiment with high density EEG using the Active-Two Biosemi EEG system (Biosemi V.O.F. Amsterdam, Netherlands) including 128 electrodes on the scalp with a sampling rate fixed at 512 Hz. All the preprocessing steps, including DC removal, filtering at 0.2 Hz (high pass) and 30 Hz (low pass), and Notch Filtering at 50 Hz to remove line current artifact, were done with the Cartool Software (Brunet et al. 2011). Each trial was inspected visually and excluded from the averaging if it was contaminated by any artifact (e.g., blinks, eye movements or noise). After visual inspection, epochs were extracted, matched in number across conditions and averaged per participant. Problematic electrodes were interpolated for each participant using 3-D splines interpolation (Perrin et al. 1987), with the same electrodes interpolated across the two uttering conditions. On average, 15.5 electrodes (range: 6–23) were interpolated per participant. Average reference was applied to the EEG data after interpolation. Eventually, we applied a spatial filter as a final step of the preprocessing procedure (see more details in Michel & Brunet 2019). Response locked epochs (i.e., aligned to the vocal onset) were extracted backwards with a time window of 175 TF (i.e., 342 ms). Epochs’ duration was selected based on the two reviews from Laganaro (2019, 2023). In the latter, it is suggested that motor speech encoding processes would take up to 300 ms of the planning time rather than the 145 ms proposed in the review of Indefrey (2011).

Electrodes’ amplitudes were compared between SS and LS with a massed approach on each electrode and time-point. This analysis was computed in the R software with the “threshold-free clusters-enhancement” (TFCE) method (Smith & Nichols 2009) using the permuco4brain R package (Frossard & Renaud 2021). This test has a high control over family-wise type I error. The analysis is based on 5000 permutation tests for repeated measure ANOVA.

The TCT (Koenig & Melie-García, 2010) aims at disentangling electrical sources from noise in the ERPs data with simple randomization techniques. In other words, this test tries to determine if the same brain topographies are obtained for a specific event with repeated measurements. Before computing topographical analyses, we applied L2 normalization of raw data to ensure investigation of qualitative differences. The TCT has been computed with the Ragu software (Koenig et al. 2011b) before performing the topographic and microstate (spatio-temporal segmentation) analyses. This step is fundamental because it constitutes a legitimate way of controlling that possible ERPs differences can be attributed to conditions and not to noise (Habermann et al. 2018).

By computing an index of dissimilarity, the TANOVA uses a non-parametric randomization test to determine at which time point ERP topographies (i.e. reference-free spatial distribution of the electric signal at scalp at a specific timepoint) significantly differ across conditions (Koenig et al. 2011a; Murray et al. 2008). The TANOVA analysis is complementary to spatio-temporal segmentation (see next analysis). Indeed, index of dissimilarity and the GFP exploits respectively the topographies and the response strength meaning that they can be measured and analyzed orthogonally (Murray et al. 2008b). A minimal duration threshold for significance can be calculated to control for the possible presence of false positives resulting from the dissimilarity analysis time point by time point (Koenig et al. 2011a).

The spatiotemporal segmentation of ERPs or microstates analysis is a two-step procedure aiming at representing conditions with several prototypical topographies or microstates maps corresponding to periods of quasi-stable topographies. Microstates analysis exploit the GFP value to decompose the signal into clusters of stable periods (60–120 ms) of electrophysiological activity (Koenig et al. 2014; Michel & Koenig 2018; Skrandies 1990). First, cluster maps were extracted from concatenated data across all conditions using the topographical atomize and agglomerate hierarchical clustering (T-AAHC) algorithm (see more in Murray et al., (2008)). These cluster maps are referred to as “template maps” and transition from one map to the other indicates changes in the global coordination of neuronal activity over time (Michel & Koenig 2018). The appropriate number of microstates maps were chosen on the basis of the cross-validation analysis provided by RAGU. During the clustering procedure, we applied a smoothing correction to get rid of sudden changes in microstates patterns due to noise. Here, 250 randomization tests of microstates models were computed to determine the best correlation between the grand mean explained variance and the number of microstates classes. Once cross-validation has been computed, the retained templates maps were fitted into participants’ individual signal for each condition in order to define the microstates and extract relevant parameters (Michel & Koenig 2018). These parameters represent several metrics of interest to describe EEG topographies: strength, timing and spatial distribution. In the present study, statistical analyses were carried out on one temporal parameter (duration, DUR) and one global measurement (area under curve, AUC) of occurrence. Note that no normalization or a-priori models have been applied during microstates analyses.

The average intensity of production was 64.60 dB for loud utterances and 51.47 dB for standard utterances. The mean intensity difference was 12.73 dB (SD = 2.95 dB; minimum (Min) = 9.46 dB; maximum (Max) = 22.94 dB) on 148 loud trials on average. Participants produced pseudowords with a global high accuracy: LS utterances were produced with a 97% accuracy rate and SS utterances with 96%. The mean production latency for LS and SS was respectively 593.22 ms (SD = 118.41 ms) and 579.74 ms (SD = 124.28 ms). The best nested linear mixed model (see Appendix B for details) demonstrated a significant main effect of the speech mode (t(7365) = 4.85, β = 15.99, standard error (SE) = 3.295, p =  < 0.01), with LS yielding longer RT as compared to SS. Furthermore, a significant interaction effect was observed between the speech mode and the experimental block by which the participants started the experiment (t(7363.254) = -2.764, SE = 4.74, p = 0.005). Particularly, post-hoc analyses using a Tukey test showed that participants who started with a LS experimental block needed an additional initialization time of 16 ms to produce LS utterances (z = -4.852, SE = 3.3, p =  < 0.001). On the contrary, the difference of estimate between LS and SS was not significant for participants who started with a SS experimental block (z = -0.851, SE = 3.4, p = 0.40).

The results of the test distribution of the TFCE procedure are presented on Fig. 2A. Amplitude differences in response-locked ERPs were observed at three time periods. From approximately -127 ms to the vocal onset (i.e., time 0), a cluster of 14 neighboring electrodes in the central-anterior region was found with lower values for the loud condition. Concerning the time period from -128 ms to-200 ms preceding the vocal onset, different amplitudes appeared on three small clusters: seven left anterior electrodes, four right anterior electrodes and five right central-parietal neighboring electrodes that tend to be more negative parietally and more positive anteriorly for the loud condition. Additionally, diverging amplitudes were observed in the -340 ms to -260 ms time period preceding the vocal onset on some sparse channels. Here, electrodes that significantly differed between the two uttering conditions had lower values for the loud condition.

Three clusters were obtained for the response-locked spatio-temporal segmentation analysis, with 97.81% of explained variance (labelled from “A” to “C” in Fig. 2D). To statistically assess the global strength and duration of each map in the participants ERPs, the template maps were fitted in participants’ individual ERPs (see details on Appendix E).

Non-parametric Friedman test (Cleophas et al. 2016) demonstrated that the map A differed significantly across conditions on the two parameters of interest (AUC: X²(1) = 8.167, p = 0.004; DUR: X²(1) = 9.783, p = 0.004). In particular, map A lasted longer in the SS condition and had higher AUC value in comparison to the loud condition. On the other hand, the microstate map C significantly differed across conditions with the loud condition entailing a higher AUC value (X²(1) = 4.55, p = 0.033) compared to the standard condition.

The material was identical to experiment 1.

The experimental procedure was similar to experiment 1, except that this time WS was the speech mode contrasted with SS. During the training session, participants were instructed to speak without vibrating their vocals folds. For those who struggled to whisper, they did several extra items while focusing on not feeling vibration in their vocal apparatus. As in experiment 1, there were 8 blocks of approximatively 50 stimuli each, 4 performed in whispered mode and 4 in standard mode in a counterbalanced order across participants.

During the offline extraction of ACC and RT, WS’ intensity was increased to 70 dB to facilitate the alignment on the vocal onset using the Praat software. The inter-judgement agreement for the ACC and the RT was respectively 94% and 90%. On the behavioral level, the mixed model contained the same variables as the first experiment. On the EEG/ERP level, the length of the response-locked ERPs across conditions was 342 (i.e., 175 TF) as in the first experiment.

The TFCE test for ERPs comparisons of amplitudes indicated no differences across conditions (see Fig. 3a) in the response-locked ERPs.

The spatio-temporal segmentation of response-locked ERPs were segmented into three microstates maps explaining 97.34% of the global explained variance (GEV). Template maps were fitted to individuals ERPs to perform two statistical analyses. As in Experiment 1, we assessed the AUC and the DUR parameters (see details in Appendix F). The non-parametric Friedman test demonstrated that neither the AUC nor the DUR differed across conditions on any of the microstates maps.