Networks and meaning
Networks and meaning
On barking dogs, sunscreen, and meaning
“The ‘meaning’ of a text is not a stable, identifiable content; it is an ongoing process of signification that is always deferred, never fully present” (Derrida (hallucinated by Claude), 1967, p. 158).
How does meaning arise?
When we look at some text, by what process do we come to ascribe meaning to it? How does language act upon us? And does synthetic language act differently?
In this blog post I’ll venture into some of these questions by looking at networks of meaning. I’ll postulate, riffing off Derrida’s Différance—the difference and deferral of meaning—that we can understand meaning making as a networked phenomenon. Here we’ll look at networks of association and predictive models that arise in the mind of the human and machine.
The standard model
In the standard model of communication, a speaker encodes a message that’s then decoded by the receiver. The message might be encoded, say, in language, that itself is upheld by social norms. De Saussure notes that the signs used in the communicative act might themselves be arbitrary—there is no inherent connection between the sign and the signified—but rather that they are upheld by social convention. This insight wasn’t novel. Aristotle himself noted that “there can be no natural connection between the sound of any language and the things signified” (Richards 1932, 32). This view raises important questions? How is social convention of language upheld? How does language shift and expand to include new signs? What is the process by which encoding and decoding happens? Where does meaning arise in the midst of this back and forth?
Decoding
How do we make sense of the sign that says “No barking dogs allowed”?
The statement immediately surfaces a number of connotations and denotations. The denotative meaning might be interpreted as:
Barking dogs should be removed from the premises.
I should probably leave with Fido.
The connotative meanings might include:
Oddly specific, only singling out barking dogs. Perhaps they don’t want to annoy the neighbors?
What other strict rules does this establishment have?
And so on…
We can map out the same denotative and connotative associations with Claude:
+-----------+
| No |
| barking |
| dogs |
| allowed |
+-----------+
|
+------------------------+------------------------+
| |
| |
+-----------+------------+ +----------+---------+
| Denotative | | Connotative |
| Associations | | Associations |
+-----------+------------+ +----------+---------+
| |
| |
+---------+--------+ +--------+-------+
| Dogs | | Noise |
| Barking | | Disturbance |
| Prohibition | | Annoyance |
| Rules | | Restriction |
| Regulations | | Control |
| Signs | | Authority |
| Public spaces | | Peace |
| Private property | | Quiet |
| Pets | | Consideration |
| Animal behavior | | Respect |
+------------------+ | Cleanliness |
| Safety |
| Discrimination |
| Exclusion |
| Elitism |
| Conformity |
+----------------+Thus the original sign “No barking dogs allowed” is mapped to yet other signs. These signs themselves have denotative and connotative associations, and so on ad infinitum. This process of infinite regress in the chain of signification is called “infinite semiosis” or the deferral of meaning. Yet we can also note that there is a structure to this relationship, one where signs map to yet other signs and where each mapping has a relationship. This is starting to shape up as a network!
Associative Networks
One way we might model chains of signification is using a network model. The simplest relation is that of association. Here “association” refers to the mental connection or link between two or more concepts, ideas, or entities. These connections can be based on various factors, such as similarity, contrast, causality, or contiguity (proximity in time or space).
Associations can be:
Denotative: Literal, direct, or explicit connections between concepts, often based on their inherent properties or definitions.
Connotative: Implied, indirect, or symbolic connections between concepts, often influenced by cultural, emotional, or personal experiences.
In an associative network, nodes represent concepts or entities, while edges or links represent the associations between them. The strength of these associations can vary, and they can be unidirectional or bidirectional.
Associative networks can be predictive. Consider for example the person that associates dark clouds with rain. When they see a dark cloud, they can, by association predict it might rain and take appropriate action. Or in the case of the “No barking dogs allowed sign” a more extensive chain of associations might occur:
They don't want barking dogs here.
I have a barking dog.
They might cause trouble if they see my dog. (association-prediction)
I don’t want trouble.
Therefore, I'll leave. (action)
This chain of associations is guided by complex cognitive processes, drawing on prior knowledge, mental-models, memory, risk avoidance, attention, inhibition, and so on. We might say these complex factors come together to form a kind of “guided activation” through the associational network. Or consider the person on the beach who decides to apply sunscreen:
Beach
Sunny weather
Sun's rays are strong
UV rays can cause sunburn & skin damage (association-prediction)
I should protect my skin
Sunscreen can help prevent sunburn (association-prediction)
I have sunscreen in my bag
I'll apply sunscreen now
This more complex example shows reasoning as a series of guided activations through the network, traversing associations and association-predictions that might be rooted in the person’s mental model of the world. These mental models might themselves be comprised of yet more associations. For example, we can decompose the prediction “UV rays can cause sunburn & skin damage” into the following associations (rendered by Claude):
UV rays
Are a type of electromagnetic radiation
Are present in sunlight
Are invisible to the human eye
Can penetrate the skin
Skin
Is the body's largest organ
Serves as a protective barrier
Contains melanin, which provides some UV protection
Can be damaged by excessive UV exposure
Sunburn
Is a type of skin damage caused by UV rays
Occurs when the skin is exposed to more UV radiation than it can handle
Causes redness, pain, and inflammation
Can increase the risk of skin cancer
Skin damage
Can be caused by various factors, including UV exposure
Includes sunburn, premature aging, and skin cancer
Can be prevented or mitigated by protective measures, such as sunscreen or clothing
UV exposure
Is the amount of UV radiation the skin is subjected to
Depends on factors such as time of day, season, latitude, and altitude
Can be reduced by seeking shade, wearing protective clothing, or using sunscreen
The person's mental model of the world, which includes knowledge about UV radiation, skin biology, sunburn, and skin damage, allows them to make this prediction based on the activation of these interconnected associations. And we could go on, decomposing these associations into yet more associations ad infinitum. It forms a sort of dizzying, recursive fractal-like pattern, with increasingly higher levels detail. Hence the need for guided activation through the network.
How are activations guided through the network? One model is meta-associations that reflect properties of “relevance,” “specificity,” “coherence,” “goal-directedness,” or "emotional salience." For example, relevance might be thought of as the association “this relates to my current situation.” Similarly, “specificity” could be modeled as the meta-association “this is specifically part of my experience.” Or other meta-associations could represent higher-order relationships between the activated associations:
“Coherence” could be modeled as the meta-association “this fits together with other related associations.”
“Goal-directedness” could be modeled as the meta-association “this helps me achieve my current objective.”
“Emotional salience” could be modeled as the meta-association “this evokes a strong emotional response.”
Returning to the beach example, we might an association network that looks like:
Beach
This relates to my current situation
This is a familiar and salient experience ↓
Sunny weather
This is specifically part of the beach experience
This relates to my current situation
This fits together with other related associations (beach, sun) ↓
Sun's rays are strong
This is a consequence of the sunny weather
This relates to my current situation
This fits together with other related associations (sunny weather, beach) ↓
UV rays can cause sunburn & skin damage
This is a specific consequence of strong sun's rays
This relates to my current situation
This fits together with other related associations (sun's rays, skin)
This evokes a negative emotional response ↓
I should protect my skin
This is a logical conclusion based on the potential for skin damage
This relates to my current situation and well-being
This fits together with other related associations (UV rays, sunburn, skin damage)
This helps me achieve the goal of avoiding harm ↓
Sunscreen can help prevent sunburn
This is a specific means of protecting the skin
This relates to my current situation and goal
This fits together with other related associations (skin protection, sunburn prevention)
This helps me achieve the goal of avoiding sunburn ↓
I have sunscreen in my bag
This is a specific instance of having access to sunscreen
This relates to my current situation and resources
This fits together with other related associations (sunscreen, skin protection)
This enables me to take action towards my goal ↓
I'll apply sunscreen now
This is a specific action I can take in the current situation
This relates to my current situation and goal
This fits together with other related associations (sunscreen, skin protection, sunburn prevention)
This helps me achieve the goal of protecting my skin
This is the logical next step based on the previous associations
In this example, the meta-associations provide a higher-order structure that guides the activation and progression of associations related to the beach experience. Notice the repetition of meta-associations like “This relates to my current situation and resources,” underscoring the ongoing importance of context. Or the way it becomes more goal directed after the negative emotional response.
From where do these meta-associations arise? One answer might be the person’s “world model,” itself a vast associative network of verbal and embodied knowledge, experiences, beliefs, and expectations about the world.
This world model is not limited to verbal or symbolic representations but also includes non-verbal, perceptual, and embodied aspects. It is shaped by the individual's sensory experiences, motor interactions, emotional responses, and social interactions, as well as by their abstract knowledge and reasoning abilities.
For example, the meta-association “This relates to my current situation” might arise not just from a verbal or conceptual understanding of the beach context, but also from the perceptual and embodied experiences of being at the beach, such as the sensation of the sun on one's skin, the sound of the waves, or the feeling of sand underfoot. These non-verbal and embodied associations contribute to the overall sense of relevance and salience that guides the activation of related associations.
Similarly, the meta-association “This evokes a negative emotional response” in relation to the potential for sunburn and skin damage might arise not just from an abstract knowledge of the harmful effects of UV rays, but also from previous embodied experiences of sunburn, such as the sensation of pain or discomfort, or the visual appearance of reddened or peeling skin. These embodied associations serve to reinforce the emotional salience and motivational significance of the related associations.
The thinker's world model can be seen as a hierarchical and multi-dimensional associative network that integrates various levels of representation, from low-level sensory and motor associations to high-level conceptual and symbolic associations. The meta-associations that guide the activation of associations in a particular context emerge from the complex interplay of these different levels of representation, reflecting the rich and dynamic nature of the individual's understanding of the world.
We can thus reconstruct the beach example using embodied meta-associations:
Beach
The feel of sand beneath my feet
The sound of waves crashing
The smell of saltwater in the air
Memories of previous beach visits ↓
Sunny weather
The warmth of the sun on my skin
Bright sunlight making me squint
The feel of a light breeze
The absence of clouds in the sky ↓
Sun's rays are strong
The intensity of the heat on my skin
Beads of sweat forming on my forehead
The urge to seek shade or cover up
The glare of the sun reflecting off the water ↓
UV rays can cause sunburn & skin damage
The memory of painful sunburns from the past
The feel of tight, hot skin after too much sun exposure
The visual image of reddened or peeling skin
The knowledge of the long-term risks of sun damage ↓
I should protect my skin
The urge to avoid the discomfort of sunburn
The desire to maintain healthy, youthful-looking skin
The memory of previous instances of sun protection
The feel of covering up with clothing or seeking shade ↓
Sunscreen can help prevent sunburn
The tactile sensation of applying sunscreen
The smell of the sunscreen lotion
The visual image of the sunscreen bottle
The memory of previous successful uses of sunscreen ↓
I have sunscreen in my bag
The feel of rummaging through my beach bag
The visual image of the sunscreen bottle among other items
The relief of knowing I have sun protection available
The anticipation of applying the sunscreen ↓
I'll apply sunscreen now
The decision to take action based on the previous associations
The motor sequence of retrieving the sunscreen and applying it
The cool, slippery feel of the lotion on my skin as I apply it
The satisfaction of knowing I'm taking steps to protect myself
The readiness to continue enjoying the beach with peace of mind
This example provides a richer and more ecologically valid representation of the beach scenario. It suggests that the “heuristics” guiding human reasoning and decision-making are deeply grounded in the thinker's sensory, motor, and affective experiences, and are shaped by their ongoing interactions with the world. The example feels more organic and perhaps more familiar with the embodied reasoning we use in the day to day.
Each of these embodied meta-associations are rooted in tactile sensations (the feel of sand, sun, breeze, sunscreen), visual images (bright sunlight, the appearance of skin damage, the sunscreen bottle), smells (saltwater, sunscreen lotion), sounds (waves crashing), and bodily sensations (warmth, sweating, discomfort, relief). The meta-associations also include memories of previous experiences (past beach visits, sunburns, successful sun protection) and anticipations of future experiences (applying sunscreen, continuing to enjoy the beach), reflecting the temporal and narrative structure of the thinker's world model.
Each of these meta associations could be said to be a result of deep cognitive and sensorial processes that make up a network of associations. The progression of associations is guided by the interplay of these embodied meta-associations, with each association building upon the sensory, affective, and motivational salience of the previous ones. The final decision to apply sunscreen emerges as a natural and adaptive response to the cumulative trace through these meta-associations, reflecting the thinker's embodied understanding of the situation and their goals.
We’ve thus seen how sensation, prediction, and action can all form naturally within the structure of an associative network. We’ve seen how predictive models of the world might themselves be a result of sets of associations—through conceptual and embodied knowledge of the world—and how those predictions can result in action (applying sunscreen). By representing sensation, prediction, and action as emergent properties of an integrated, embodied associative network, this framework offers a parsimonious explanation of how humans make sense of and respond to their world.
Properties of Associative Networks
Now that I’ve given a provisional account—a napkin sketch of sorts—of associative networks, it’s perhaps worth stepping back and thinking about their behavior.
Meaning is deferred and always in flux.
Drawing on Derrida’s neologism différance, which combines the meanings of “difference” and “deferral,” suggests that the meaning of a sign is never fully present or fixed but is always contingent upon its relationship to other signs in a system of signification. According to Derrida, the meaning of a sign is constantly deferred, as it depends on a chain of signifiers that never reaches a final, transcendental signified.
Similarly, the associative network model proposes that the meaning of an sentence or any other semiotic object emerges from the complex web of associations it activates in the reader’s mind. These associations are not fixed or inherent in the sentence itself but are shaped by the reader’s mental models (including their cultural, historical, and personal context), as well as by the relationships between the sentence and other signs in the semiotic system.
Both accounts give a view of meaning that’s not a static, stable structure, but one that’s context-dependent, dynamic, and always in flux. In the associative network model, the meaning of an sentence is not a static property but a fluid and emergent phenomenon that arises from the interplay of multiple associative pathways. Similarly, Derrida's différance suggests that meaning is always in a state of flux, constantly deferred and differed through the endless play of signifiers. This means associative networks are highly volatile, context-dependent structures that can be traversed with potentially endless, non-determinate outcomes.
Associative networks in language models
Similarly, when language models generate text, they might themselves be activating associational networks amidst their inner structures. There is some evidence to support this, for example in Hernandez et al.1 they describe the pervasive role of relations within language models, a subset of which are “well-approximated by a single linear transformation on the subject representation.” They go on to note that such first order approximations “exist for a variety of factual, commonsense, and linguistic relations.” And generalize more broadly to claim that “Much of the knowledge encoded in transformer language models (LMs) may be expressed in terms of relations: relations between words and their synonyms, entities and their attributes, etc.” Such relations may be seen as specific cases of associations within the language model. Indeed, the authors find that “LM predictions capture relational knowledge accurately, but this knowledge is not linearly encoded in their representations.”
Indeed, in my own probings with language models, I’ve observed that they are adept at creating associative networks given simple prompts. As we saw with some of the Claude generated association networks above, the model can pinpoint relevant associations. I’ve experimented with creating vast association graphs using the language model to better understand the contours of the embedded knowledge they surface. One early association graph you can access here (I’ve linked to the node “Rhizome” in the association network). Try clicking between nodes and see what you observe. In moving with hyperlinks within the network’s structure, you’re “activating” nodes in the wider associational network. You can challenge yourself to see how far you can go—there are nearly 10,000 nodes in the network.
I’ll leave my own explorations of synthetic association networks to a future Substack. But for now, I’ll reflect on the differences between human and machine generated associational networks. While their structures and content might be superficially similar, under the hood very different processes of activation are occurring. Human associational networks are psychological and neural, whereas computational approaches rely on vast networks of mathematical operations—artificial neurons. Another key difference is the role of context and embodiment. Language models don’t have access to the same sensory information that guide human associational networks, nor do they have as deep a cultural and social context to guide them. Instead they’re trained on a vast corpus of data that determines, in part, what associations they are able to draw. This means that while human associations might be more embodied, synthetic ones might contain more “facts,” scientific associations, and common tropes surfaced by gathering patterns in the data (this is still a provisional account, based on my own observations).
The differences between human and synthetic associational networks has implications for how we interpret AI-generated content. While language models can produce impressively coherent and often insightful text, their lack of embodied experience and cultural grounding means their associations may sometimes be shallow or miss important nuances that humans intuitively grasp. Conversely, their ability to process vast amounts of textual data allows them to make connections that might not be immediately apparent to humans, potentially offering novel insights or perspectives. This interplay between human and machine-generated associations opens up intriguing possibilities for augmenting human creativity and problem-solving. By leveraging the strengths of both systems—the depth and embodied nature of human associations combined with the breadth and pattern-recognition capabilities of machine learning—we might be able to explore new possibilities of human-machine symbiosis.
That the internal structure of language models may have associational networks embedded within them raises a number of questions:
How are these internal structures represented, from the associations inherent in neurons to the larger circuits that they form within the network?
How can AI probes, such as generating associational networks, reveal these internal structures and their properties?
How does the presence of associational networks in language models support our understanding of resonance?
How do human and machine associational networks interplay when traversing the semioscape? What possibilities does this open up for expanding human-AI collaboration?
To what extent can language models form novel associations not explicitly present in their training data? How does this compare to human creativity and innovation?
How do associational networks in multilingual language models differ from those in monolingual models? What can this tell us about the relationship between language and conceptual structure?
We’ll explore some of these questions and others in future Substack posts. For now, I want to return to the empirical basis for associational networks (with some help from Claude).
Empirical Support for Associative Networks
We might look at associational networks not merely as a theoretical construct, but as one supported by a wealth of empirical evidence from cognitive science, neuroscience, and psycholinguistics. This evidence not only corroborates the existence of associative networks but also illuminates how they function in human cognition.
Foundational work by Collins and Loftus (1975)2 on spreading activation in semantic memory provides a cornerstone for understanding how concepts are linked in our minds. Their research demonstrated that when a concept is activated, this activation spreads to related concepts, much like our proposed model of associative networks. This spreading activation manifests in phenomena such as semantic priming, where exposure to one word facilitates the recognition of related words. Meyer and Schvaneveldt's (1971)3 classic study showed that people recognize words more quickly when they are preceded by semantically related words, providing clear evidence for the interconnected nature of our mental lexicon.
The neural basis for these associative networks has been further elucidated by modern neuroimaging techniques. Huth et al.'s (2016)4 groundbreaking fMRI study mapped how the brain represents semantic information, revealing that semantically related concepts activate similar brain regions. This neural evidence strongly supports the idea that our conceptual knowledge is organized in a network-like structure, where related concepts are physically proximate in the brain's architecture.
Our discussion of embodied meta-associations finds support in the field of embodied cognition. Barsalou's (1999)5 perceptual symbol systems theory provides a framework for understanding how conceptual knowledge is grounded in sensory-motor systems. This aligns with our proposal that meta-associations like “The feel of sand beneath my feet” or “The warmth of the sun on my skin” play a crucial role in guiding activations through the associative network. The embodied nature of cognition suggests that our conceptual systems are deeply intertwined with our sensory and motor experiences, supporting the idea that meaning-making is a holistic process involving both abstract and concrete associations.
The fluid and context-dependent nature of meaning in associative networks, reminiscent of Derrida's différance, finds empirical support in studies of flexible conceptual processing. Yee and Thompson-Schill's (2016)6 work demonstrates how the meaning of concepts can shift based on context. This malleability of meaning aligns with our proposition that meaning in associative networks is not fixed but emerges from the dynamic interplay of associations activated in a given context.
The predictive aspect of associative networks, as illustrated in our beach example, resonates with contemporary theories of brain function. Clark's (2013)7 work on predictive processing provides a neuroscientific framework for understanding how prediction and action emerge from hierarchical generative models in the brain. This aligns with our view of associative networks as not just passive repositories of knowledge, but active predictive models that guide behavior and decision-making.
Finally, the cultural mediation of associative networks is supported by cross-cultural cognitive research. Studies like those conducted by Ji et al. (2004)8 demonstrate how cultural background influences cognitive processes and associative thinking. This empirical evidence underscores our claim that the structure and content of associative networks are shaped by cultural, historical, and personal contexts.
These empirical findings from various domains of cognitive science provide robust support for the associative network theory of meaning. They demonstrate that our proposed model is not just a philosophical construct, but a reflection of how the human mind might process and create meaning. The evidence suggests that meaning indeed emerges from a complex, dynamic network of associations, shaped by embodied experience, cultural context, and ongoing interactions with the world. This empirical grounding strengthens our argument that meaning is not a static property of signs, but a fluid and emergent phenomenon arising from the intricate dance of associations in our minds.
I hope you find it useful.
Hernandez, Evan, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. "Linearity of relation decoding in transformer language models." arXiv preprint arXiv:2308.09124 (2023).
Collins, A. M., & Loftus, E. F. (1975). A spreading-activation theory of semantic processing. Psychological Review, 82(6), 407-428.
Meyer, D. E., & Schvaneveldt, R. W. (1971). Facilitation in recognizing pairs of words: Evidence of a dependence between retrieval operations. Journal of Experimental Psychology, 90(2), 227–234. https://doi.org/10.1037/h0031564
Huth, A., de Heer, W., Griffiths, T. et al. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532, 453–458 (2016). https://doi.org/10.1038/nature17637
Barsalou LW. Perceptual symbol systems. Behav Brain Sci. 1999 Aug;22(4):577-609; discussion 610-60. doi: 10.1017/s0140525x99002149. PMID: 11301525.
Yee, E., Thompson-Schill, S.L. Putting concepts into context. Psychon Bull Rev 23, 1015–1027 (2016). https://doi.org/10.3758/s13423-015-0948-7
Clark A. Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behav Brain Sci. 2013 Jun;36(3):181-204. doi: 10.1017/S0140525X12000477. Epub 2013 May 10. PMID: 23663408.
Ji, L.-J., Zhang, Z., & Nisbett, R. E. (2004). Is It Culture or Is It Language? Examination of Language Effects in Cross-Cultural Research on Categorization. Journal of Personality and Social Psychology, 87(1), 57–65. https://doi.org/10.1037/0022-3514.87.1.57