Early language exposure affects neural mechanisms of semantic representations
Curation statements for this article:-
Curated by eLife
eLife assessment
This study provides important evidence regarding the development of concept representations, using functional brain imaging to compare concept structure in people with different amounts of language experience. The analyses, which are overall solid, suggest that representations in the left lateral anterior temporal lobe differ as a function of childhood language experience.
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
- Neuroscience (eLife)
Abstract
One signature of the human brain is its ability to derive knowledge from language inputs, in addition to nonlinguistic sensory channels such as vision and touch. How does human language experience modulate the mechanism by which semantic knowledge is stored in the human brain? We investigated this question using a unique human model with varying amounts and qualities of early language exposure: early deaf adults who were born to hearing parents and had reduced early exposure and delayed acquisition of any natural human language (speech or sign), with early deaf adults who acquired sign language from birth as the control group that matches on nonlinguistic sensory experiences. Neural responses in a semantic judgment task with 90 written words that were familiar to both groups were measured using fMRI. The deaf group with reduced early language exposure, compared with the deaf control group, showed reduced semantic sensitivity, in both multivariate pattern (semantic structure encoding) and univariate (abstractness effect) analyses, in the left dorsal anterior temporal lobe (dATL). These results provide positive, causal evidence that language experience drives the neural semantic representation in the dATL, highlighting the roles of language in forming human neural semantic structures beyond nonverbal sensory experiences.
Article activity feed
-
-
Author Response
Reviewer #1 (Public Review):
In this study, the authors set out to determine the degree to which early language experience affects neural representations of concepts. To do so, they use fMRI to measure responses to 90 words in adults who are deaf. One group of deaf adults (n=16) were native signers (and thus had early language exposure); a second group (n=21) was exposed to sign language later on. The groups were relatively well-matched in other respects. The primary finding was that the high dimensional representations of concepts in the left lateral anterior temporal lobe (ATL) differed between native and delayed signers, suggesting a role for early language experience in concept representation.
The analyses are carefully conducted and reflect a number of thoughtful choices. These include the "inverted MDS" method …
Author Response
Reviewer #1 (Public Review):
In this study, the authors set out to determine the degree to which early language experience affects neural representations of concepts. To do so, they use fMRI to measure responses to 90 words in adults who are deaf. One group of deaf adults (n=16) were native signers (and thus had early language exposure); a second group (n=21) was exposed to sign language later on. The groups were relatively well-matched in other respects. The primary finding was that the high dimensional representations of concepts in the left lateral anterior temporal lobe (ATL) differed between native and delayed signers, suggesting a role for early language experience in concept representation.
The analyses are carefully conducted and reflect a number of thoughtful choices. These include the "inverted MDS" method for constructing semantic RDMs, a normal hearing comparison group for both behavioral and fMRI data, and care taken to avoid bias in defining functional ROIs. And, comparing early and delayed signing groups is a clever way to study the role of early language experience on adult language representations.
We greatly appreciate the reviewer’s positive evaluation and constructive comments on our study.
One interesting result that I struggled to put in a broader context relates to the disconnect between behavioral and neural results. Specifically, the behavioral semantic RDMs (Figure 1a) did not differ between any of the groups of participants. This suggests that the representations of the 90 concepts are represented similarly in all of the participants. However, the similarity of the neural RDMs in left lateral ATL differs between the native and delayed signing groups (but not in other regions). Given the similarity of the behavioral semantic RDMs, it is unclear how to interpret the difference in left lateral ATL representations. In other words, the neural differences in left ATL do not affect behavior (semantic representation). The importance of the differences in neural RDMs is therefore questionable.
Thank you for this comment. In the Revision we have added explicit discussions about this important issue of the relationship between the behavioral and neural profiles for semantics:
Introduction (pages 4-5): “(previous) studies have reported little effects on semantics behaviors, including semantic interference effects in the picture-sign paradigm (Baus et al., 2008), scalar implicature (Davidson and Mayberry, 2015), or accuracy scores of several written word semantic tasks (e.g., synonym judgment) (Choubsaz and Gheitury, 2017). However, as shown by the color knowledge in the congenitally blind studies (e.g., Wang et al., 2020), similar semantic behaviors may arise from (partly) different neural representations. Semantic processing is supported by a multifaceted cognitive system and a complex neural network entailing distributed semantic regions (Bi, 2021; Binder and Desai, 2011; Lambon Ralph et al., 2017; Martin, 2016), and thus focal neural changes may not necessarily lead to semantic behavioral changes. Neurally, neurophysiological signatures assumed to reflect semantic processes showed incongruent effects across studies: N400 effects in the semantic violation of written sentences were not affected (Skotara et al., 2012), whereas M400 in the picture-sign matching task showed atypical activation patterns (reduced recruitment of left fronto-temporal regions and involvement of right parietal and occipital regions) (Ferjan Ramirez et al., 2016, 2014; Mayberry et al., 2018). It remains to be tested whether and where delayed L1 acquisition affects how semantics are neurally represented, using imaging techniques with higher spatial resolutions.”
Discussion (pages 17-18): “Notably, different from phonological and syntactic processes, where both visible behavioral underdevelopment (e.g., Caselli et al., 2021; Cheng and Mayberry, 2021; Mayberry et al., 2002) and brain functional changes (Mayberry et al., 2011; Richardson et al., 2020; Twomey et al., 2020) were observed, for semantics we only observed brain functional changes in dATL but no visible behavioral effects. Consistent with the literature where deaf delayed signers did not show differences to controls in semantic interference effects in the picture-sign paradigm (Baus et al., 2008), scalar implicature (Davidson and Mayberry, 2015), or N400 measures (Skotara et al., 2012), we did not observe visible differences in terms of semantic distance structures (Figure 1a) or reaction time of lexical decision and word-triplet semantic judgment (Supplementary file 1). As reasoned in the Introduction, this seeming neuro-behavior discrepancy might be related to the multifaceted, distributed nature of the cognitive and neural basis of semantics more broadly. The general semantic behavioral tasks we employed could be achieved with representations derived from multiple types of experiences, supported by highly distributed neural systems (e.g., (Bi, 2021; Binder and Desai, 2011; Lambon Ralph et al., 2017; Martin, 2016), including those not affected by the delayed L1 acquisition in regions beyond the dATL. This finding invites future studies to specify the exact developmental mechanisms in the left dATL (Fu et al., 2022; Unger and Fisher, 2021) and to uncover semantic behavioral consequences related to the functionality of this area.”
An important point is that, if I understand correctly, the semantic space is defined by the 90 experimental items. That is, behavioral RDMs were created by having normal hearing participants arrange 90 items spatially, and neural RDMs were created by comparing patterns of responses to these 90 experimental items. This 90-dimensional space is thus both (a) lower dimensional than many semantic space models that include hundreds of directions and (b) constrained by the specific 90 experimental items chosen. On the one hand, this seems to limit the generalizability of the findings for semantic representations more broadly.
Indeed, for the RDM the spaces were constructed by the relations among the 90 items, as is the standard practice for current RSA analyses. Regarding the dimensionality issue, we would like to clarify that although the space is a 90 x 90 matrix, the semantic distance for each pair was obtained by the subjects’ ratings, i.e., the psychological space, which is likely to be high-dimensional in nature. That is, we compressed the potentially high-dimensional psychological construct into one measure to construct the 90 x 90 matrix. If we understood correctly, semantic space models with hundreds of directions the reviewer referred to are various types of embedding and/or distributional models. There although each word is projected onto a high-dimensional vector, the distance for each pair is still extracted (e.g., by cosine similarity) to construct the cross-item similarity matrix for RSA. Regarding the generalization of the findings across items, we greatly appreciate this concern and indeed that was one of the reasons why we extracted the categorical structure based on the clustering of the items (see also response to the next Comment). We also examined the univariate abstractness contrast, which looked at the broad categorical effects rather than specific items. We have made clarifications accordingly in the Revision to address these concerns (page 8).
The logic behind using a categorical semantic RDM (e.g., Figure 2a) was not clear. The behavioral semantic RDMs (Figure 1a) clearly show gradations in dissimilarity, particularly for the abstract categories. It would seem that using the behavioral semantic RDM would capture a more accurate representation of the semantic space than the categorical one.
Thank you for this suggestion. We opted for the categorical structural similarity based on the clustering analyses to boost signal and to allow for better generalization across items (i.e., along the categorical structure). Agreeing with the reviewer that such an approach may lose the important graded space especially for the abstract items, we added an analysis using continuous semantic distances specifically focused on the abstract items (page 10):
“1) Types of semantic distance measures: While semantic categories for concrete/object words are robust and well-documented, the semantic categorization within the abstract/nonobject words is much fuzzier and remains controversial (Catricalà et al., 2014; Wang et al., 2021). The behavioral semantic RDM in Figure 1a indeed shows gradations in dissimilarity for abstract/nonobject words. We thus checked the two groups’ semantic RDMs using the continuous behavioral measures and further examined whether group differences in the left dATL were affected by the types of semantic distance (categorical vs. continuous) being used for abstract/nonobject words. The two deaf groups showed comparable similarities to the hearing benchmark (by correlating each deaf subject’s RDM with the group-averaged RDM of hearing subjects, Welch’s t23.0 = -0.12, two-tailed p = .90). RSA was performed by correlating each deaf subject’s neural RDM in the left dATL with these two types of semantic RDMs. Significant group differences were observed (Figure 3), for both the categorical RDM (Welch’s t31.0 = 3.06, two-tailed p = .005, Hedges’ g = 0.98) and the continuous behavioral semantic RDM (Welch’s t36.7 = 2.47, two-tailed p = .018, Hedges’ g = 0.76), with significant semantic encoding in dATL observed in both analyses for native signers (one-tailed ps < .003) and neither for delay signers (one-tailed ps > .42). These results indicate that the reduced dATL encoding of abstract/nonobject word meanings induced by delayed L1 acquisition was reliable across semantic distance measures.”
As the reviewer suggested, we could also carry out RSA using the 90-word behavioral semantic RDM. We did observe similar group differences with this RDM, with delayed signers showing a trend of semantic encoding reduction in the left dATL relative to native signers (native signers, mean (SD): 0.019 (0.023); delayed signers, mean (SD): 0.006 (0.022), Welch’s t31.5 = 1.78, two-tailed p = .085; a delayed signer was excluded from this analysis for being an outlier beyond 3 standard deviations). It appears that the behavioral semantic RDM yielded smaller effect sizes in group differences than the categorical RDM, but the ANOVA (the within-subject factor - RDM-type: categorical, behavioral; the between-subject factor – group: native, delayed) revealed no significant effects of RDM-type or its interaction with the group (ps > .71), but a significant main effect of group (F(1,36) = 9.19, p = .004). The seemingly weaker group differences using the behavioral semantic RDM should not be over-interpreted.
Reviewer #2 (Public Review):
The authors investigated patterns of fMRI activation for familiar words in two groups of deaf people. One "language rich" group received exposure to sign from birth, whereas the "language poor" group included kids born to hearing parents who had limited exposure to language during the first few years of life. The primary findings involved group differences in BOLD activation patterns across different areas of interest within the semantic network when participants made intermittent 1-back category judgments for words appearing in succession.
There was much to be liked about this study, including the rigor of the methods and the novel contrasts of two deaf samples. These strengths were balanced by a number of questions about the assumptions and theoretical interpretations underlying the data. I will elaborate on the major points in the paragraphs to follow, but briefly, the ways in which the authors are framing critical period constraints in language fundamentally differ from the standard nativist perspectives (e.g., Chomsky, Lenneberg). The assumptions of what constitutes a deprivation model require further justification and perhaps recasting to avoid unnecessary stigma (i.e., this reviewer was uncomfortable with the assertion that being born deaf to hearing parents by default constitutes deprivation). The introduction lacked principled hypotheses that motivated the choice of comparing abstract and concrete words, and potential accounts of group differences were underdeveloped (e.g., how do parents in China typically react to having a deaf child, and what supports are in place for preventing language deprivation? Are newborn infants universally screened for hearing loss in China? The answers to these questions might help the readers to understand why/how deaf children in this circumstance might experience deprivation).
We appreciate the reviewer’s positive evaluations and constructive comments on our study. We have revised the manuscript substantially in light of these comments (see below).
References to critical periods require a bit more elaboration with respect to lexical-semantic vs. semantic acquisition. The nature of the critical period in language acquisition remains controversial with respect to its constraints. Lenneberg and Chomsky speculated that the limit of the critical period for language acquisition was about puberty (13ish years of age). This is much older than the deaf sample tested here so arguments about aging out of the critical period at least for language acquisition need more nuance. Another issue relates to learning semantic mappings vs. learning language as falling under the same critical period umbrella. This seems highly unlikely as semantic acquisition in early childhood is aided by linguistic labeling but would likely occur in parallel even in the context of language deprivation. Much of the prior literature on critical periods and nativist approaches to language development has focused on syntactic acquisition and elements such as recursion rather than a mapping of symbols to conceptual referents. This makes the critical period group comparison somewhat tenuous because what you are really interested in is a critical period for word meaning acquisition not the more general case of syntactic competency.
The point above is highlighted in the following statement underlying one of the primary assumptions of the study:
Pg. 3, "Here, we take advantage of a special early-life language-deprivation human model: individuals who were born profoundly deaf in hearing families and thus had very limited natural language exposure (speech or sign) during the critical period of language acquisition in early childhood"
"hypofunction of the language system as a result of missing the critical period of language acquisition" (pg 3), same critique as previous - the critical period window is thought to be 13ish years old.
There are a couple of problems with this assertion/assumption. Although it is true that most children who are born deaf have hearing parents, it is not justifiable to label this condition an early-life deprivation model. Hearing parents who are extremely motivated to learn sign language and pursue related language enrichment strategies can successfully offset many of these effects. Similarly, it is not inconceivable that a deaf child born to a deaf parent might be neglected or abandoned without the benefit of early sign exposure. My argument here is that classifying deaf children born to hearing parents as automatically 'language deprived' is potentially both stigmatizing and scientifically unjustified.
We originally used the term “language deprivation” because it has been recently advocated in the deaf field mainly to increase society’s awareness of the risks of language deprivation and the lifelong impact that deaf and hard-of-hearing children face (e.g., Hall, 2017, Maternal and Child Health Journal; Lillo-Martin & Henner, 2020, Annual Review of Linguistics). In the current context, we agree with the reviewer that “early-life deprivation” model may not precisely describe the language acquisition condition of delayed signers. Indeed, for some of the delayed subjects in our study, their hearing parents actively tried to provide additional aids of exposure to signs (via preschool special education programs; learning signs by themselves) or speech (via hearing aids). In the revision, we avoided the term “language deprivation” and used the terms “subjects with varying amounts and qualities of early language exposure” or “delayed L1 acquisition” to more precisely describe our experimental manipulation throughout the revised manuscript.
We fully agree with the reviewer that the “critical period” of language acquisition is too much an umbrella term, which may be taken to refer to critical period for different, specific cognitive and/or neural development in the literature. In the Revision we avoided using this term to reduce ambiguity. Instead, we now made explicit throughout the specific processes being discussed (phonology, syntax, semantics). The effects of early language experience (reduced in delayed L1 acquisition) on the behavioral and neural patterns relating to phonology, syntax, and semantics are now elaborated, discussed separately and explicitly in both the Introduction and Discussion (pages 3-4, 17-18).
Regarding the potential nonlinguistic socio-environmental differences (e.g., coping strategies after deafness awareness), we have added further clarifications (page 15): “Notably, routine nation-wide neonate hearing screening in China did not start until 2009, years after the early childhood of our participants (born before 2000), and some hearing parents may nonetheless try to give deaf children additional aids of exposure to signs (via preschool special education programs) or speech (via hearing aids). Critically, our positive results of the robust group differences in dATL suggest that early homesign/aid measures and later formal education for sign and written language experiences are insufficient for typical dATL neurodevelopment; the full-fledged language experience during early infancy and childhood (before school age) plays a necessary role in this process.” Relevant information has also been added in the Method/Result sections.
Pg. 6 "It should be noted that the neural semantic abstractness effect does not equate with language-derived semantic knowledge, as it might arise from some nonverbal cognitive processes that are more engaged in abstract word processing (Binder et al., 2016)." - I had great difficulty understanding what this meant.
We have revised this sentence as follows: “While the abstractness effect has often been used to reflect linguistic processes (e.g., (Wang et al., 2010)), “abstractness” is not a single dimension and instead relates to both linguistic and nonlinguistic (e.g., emotion) cognitive processes (Binder et al., 2016; Troche et al., 2014; Wang et al., 2018).” (page 11)
-
eLife assessment
This study provides important evidence regarding the development of concept representations, using functional brain imaging to compare concept structure in people with different amounts of language experience. The analyses, which are overall solid, suggest that representations in the left lateral anterior temporal lobe differ as a function of childhood language experience.
-
Reviewer #1 (Public Review):
In this study, the authors set out to determine the degree to which early language experience affects neural representations of concepts. To do so, they use fMRI to measure responses to 90 words in adults who are deaf. One group of deaf adults (n=16) were native signers (and thus had early language exposure); a second group (n=21) was exposed to sign language later on. The groups were relatively well-matched in other respects. The primary finding was that the high dimensional representations of concepts in the left lateral anterior temporal lobe (ATL) differed between native and delayed signers, suggesting a role for early language experience in concept representation.
The analyses are carefully conducted and reflect a number of thoughtful choices. These include the "inverted MDS" method for constructing …
Reviewer #1 (Public Review):
In this study, the authors set out to determine the degree to which early language experience affects neural representations of concepts. To do so, they use fMRI to measure responses to 90 words in adults who are deaf. One group of deaf adults (n=16) were native signers (and thus had early language exposure); a second group (n=21) was exposed to sign language later on. The groups were relatively well-matched in other respects. The primary finding was that the high dimensional representations of concepts in the left lateral anterior temporal lobe (ATL) differed between native and delayed signers, suggesting a role for early language experience in concept representation.
The analyses are carefully conducted and reflect a number of thoughtful choices. These include the "inverted MDS" method for constructing semantic RDMs, a normal hearing comparison group for both behavioral and fMRI data, and care taken to avoid bias in defining functional ROIs. And, comparing early and delayed signing groups is a clever way to study the role of early language experience on adult language representations.
One interesting result that I struggled to put in a broader context relates to the disconnect between behavioral and neural results. Specifically, the behavioral semantic RDMs (Figure 1a) did not differ between any of the groups of participants. This suggests that the representations of the 90 concepts are represented similarly in all of the participants. However, the similarity of the neural RDMs in left lateral ATL differs between the native and delayed signing groups (but not in other regions). Given the similarity of the behavioral semantic RDMs, it is unclear how to interpret the difference in left lateral ATL representations. In other words, the neural differences in left ATL do not affect behavior (semantic representation). The importance of the differences in neural RDMs is therefore questionable.
An important point is that, if I understand correctly, the semantic space is defined by the 90 experimental items. That is, behavioral RDMs were created by having normal hearing participants arrange 90 items spatially, and neural RDMs were created by comparing patterns of responses to these 90 experimental items. This 90-dimensional space is thus both (a) lower dimensional than many semantic space models that include hundreds of directions and (b) constrained by the specific 90 experimental items chosen. On the one hand, this seems to limit the generalizability of the findings for semantic representations more broadly.
The logic behind using a categorical semantic RDM (e.g., Figure 2a) was not clear. The behavioral semantic RDMs (Figure 1a) clearly show gradations in dissimilarity, particularly for the abstract categories. It would seem that using the behavioral semantic RDM would capture a more accurate representation of the semantic space than the categorical one.
-
Reviewer #2 (Public Review):
The authors investigated patterns of fMRI activation for familiar words in two groups of deaf people. One "language rich" group received exposure to sign from birth, whereas the "language poor" group included kids born to hearing parents who had limited exposure to language during the first few years of life. The primary findings involved group differences in BOLD activation patterns across different areas of interest within the semantic network when participants made intermittent 1-back category judgments for words appearing in succession.
There was much to be liked about this study, including the rigor of the methods and the novel contrasts of two deaf samples. These strengths were balanced by a number of questions about the assumptions and theoretical interpretations underlying the data. I will elaborate …
Reviewer #2 (Public Review):
The authors investigated patterns of fMRI activation for familiar words in two groups of deaf people. One "language rich" group received exposure to sign from birth, whereas the "language poor" group included kids born to hearing parents who had limited exposure to language during the first few years of life. The primary findings involved group differences in BOLD activation patterns across different areas of interest within the semantic network when participants made intermittent 1-back category judgments for words appearing in succession.
There was much to be liked about this study, including the rigor of the methods and the novel contrasts of two deaf samples. These strengths were balanced by a number of questions about the assumptions and theoretical interpretations underlying the data. I will elaborate on the major points in the paragraphs to follow, but briefly, the ways in which the authors are framing critical period constraints in language fundamentally differ from the standard nativist perspectives (e.g., Chomsky, Lenneberg). The assumptions of what constitutes a deprivation model require further justification and perhaps recasting to avoid unnecessary stigma (i.e., this reviewer was uncomfortable with the assertion that being born deaf to hearing parents by default constitutes deprivation). The introduction lacked principled hypotheses that motivated the choice of comparing abstract and concrete words, and potential accounts of group differences were underdeveloped (e.g., how do parents in China typically react to having a deaf child, and what supports are in place for preventing language deprivation? Are newborn infants universally screened for hearing loss in China? The answers to these questions might help the readers to understand why/how deaf children in this circumstance might experience deprivation).
References to critical periods require a bit more elaboration with respect to lexical-semantic vs. semantic acquisition. The nature of the critical period in language acquisition remains controversial with respect to its constraints. Lenneberg and Chomsky speculated that the limit of the critical period for language acquisition was about puberty (13ish years of age). This is much older than the deaf sample tested here so arguments about aging out of the critical period at least for language acquisition need more nuance. Another issue relates to learning semantic mappings vs. learning language as falling under the same critical period umbrella. This seems highly unlikely as semantic acquisition in early childhood is aided by linguistic labeling but would likely occur in parallel even in the context of language deprivation. Much of the prior literature on critical periods and nativist approaches to language development has focused on syntactic acquisition and elements such as recursion rather than a mapping of symbols to conceptual referents. This makes the critical period group comparison somewhat tenuous because what you are really interested in is a critical period for word meaning acquisition not the more general case of syntactic competency.
The point above is highlighted in the following statement underlying one of the primary assumptions of the study:
Pg. 3, "Here, we take advantage of a special early-life language-deprivation human model: individuals who were born profoundly deaf in hearing families and thus had very limited natural language exposure (speech or sign) during the critical period of language acquisition in early childhood""hypofunction of the language system as a result of missing the critical period of language acquisition" (pg 3), same critique as previous - the critical period window is thought to be 13ish years old.
There are a couple of problems with this assertion/assumption. Although it is true that most children who are born deaf have hearing parents, it is not justifiable to label this condition an early-life deprivation model. Hearing parents who are extremely motivated to learn sign language and pursue related language enrichment strategies can successfully offset many of these effects. Similarly, it is not inconceivable that a deaf child born to a deaf parent might be neglected or abandoned without the benefit of early sign exposure. My argument here is that classifying deaf children born to hearing parents as automatically 'language deprived' is potentially both stigmatizing and scientifically unjustified.
Pg. 6 "It should be noted that the neural semantic abstractness effect does not equate with language-derived semantic knowledge, as it might arise from some nonverbal cognitive processes that are more engaged in abstract word processing (Binder et al., 2016)." - I had great difficulty understanding what this meant.
-
Reviewer #3 (Public Review):
This work extends earlier findings from this group which showed in congenitally blind individuals preserved, presumably language-derived, representations of colour knowledge are present only in dATL. While the present study confirms the importance of language in representations in dATL, the specificity of dATL hinges on descriptive rather than inferential statistics, and future studies may be needed to demonstrate the primacy of dATL in language-based representation as well as the generalisability of effects across different flavours of conceptual knowledge.
-
