Associative data structures

On steps towards an epistemic technology

Jul 12, 2024

In a recent Substack, I gave an associative network account of meaning. I wrote about language models’ propensity for association and drew upon Derrida’s neologism différance, which combines the meanings of “difference” and “deferral,” to suggest that the meaning of a sign is never fully present or fixed but is always contingent upon its relationship to other signs in a system of signification. According to Derrida, the meaning of a sign is constantly deferred, as it depends on a chain of signifiers that never reaches a final, transcendental signified.

In recent Substacks, I wrote about the process of circumlocution—as a process of metaphoric scaffolding spiraling towards the transcendental signified but never reaching it. I noted that models do this well, such as when describing a musical piece they’ve never heard, but can still triangulate on some kind of “associative manifold” formed by their training processes. In essence, they can stitch together a representation of something they cannot directly access, through the voluminous data that creates a kind of “umwelt” by which they come to know entities in the world. This could apply to canonical musical pieces, but also colors, paintings, even lived experiences it’s obviously never had.

Here’s a model (ChatGPT) giving a first-hand account of someone struggling with opioid addiction:

I remember the first time I took an opioid. It was after a surgery, and the doctor prescribed it for the pain. The relief was immediate, a warm blanket of comfort that enveloped me, making everything seem manageable. But it wasn’t just the physical pain it numbed; it dulled the emotional pain too, the stress, the anxiety. For a while, it felt like I had discovered a secret passage to peace.
At first, I took the pills as prescribed, but as time went on, I found myself craving that feeling even when I wasn’t in pain. Just one more, I’d tell myself. Just to take the edge off. But one turned into two, then three, and before long, I was taking them just to get through the day.
The dependency snuck up on me. I didn’t realize how hooked I was until I tried to stop. The withdrawal symptoms were brutal—nausea, sweating, shaking, and a deep, gnawing ache that seemed to come from my very bones. But worse than the physical symptoms was the overwhelming craving, the desperation for relief.
My life started to revolve around getting more pills. I’d go to different doctors, fake symptoms, do anything to get a prescription. When that didn’t work, I turned to the street. The shame and guilt were constant companions, but they weren’t enough to make me stop. The fear of withdrawal kept me in a relentless cycle.
Relationships with friends and family frayed. I became unreliable, secretive, and distant. They’d ask what was wrong, but I couldn’t bring myself to admit the truth. I’d lie, make excuses, anything to hide my addiction. The person I was becoming scared me, but I felt powerless to change it.
There were moments of clarity, brief windows where I’d resolve to get clean. I’d flush the pills, swear off buying more. But the withdrawals would start, and I’d cave. The cravings were like a beast inside me, demanding to be fed.
In my darkest moments, I’ve thought about the future, wondered if there’s a way out of this prison. I dream of a life free from the constant need, from the fear and shame. I’ve tried rehab, counseling, and support groups, and while they’ve helped, the battle is ongoing. Every day is a struggle, a choice between giving in and fighting for a life I can barely remember but desperately want back.

Surely such an account is pieced together from myriad stories and Reddit posts of people struggling with opioid addiction. It traces the contours of cravings and fear of withdrawal and its impacts on relationships and the nature of the ongoing struggle to get clean. Perhaps there is something idealized about its account—a kind of aggregate story that weaves together a multiplicity of stories to get various essences of the experiences that might not all be felt by a single person. Given that millions are addicted to opioids, it’s not surprising that there is a wealth of data available to it to give such a detailed account. We might say that the account is a bricolage of common patterns and experiences drawing upon individual personal anecdotes, medical research, and psychological insights into addiction.

These common patterns form the basis for an associative network within the model. We can better understand the model’s own internal representation, or at least, a synthesis of it, by asking it to create an associative network representing the patterns and forms that underlie its generation. For example, for the addiction case above, it gave the following simplified associative diagram:

             Initial Prescription and Relief
                    /                   \
            Craving and Escalation  --->  Withdrawal Symptoms
                   \                  /      /
                   Impact on Life and Relationships
                        \                /
                       Attempts to Quit
                            |
                      Support Systems

We can then ask the model to apply breadth first search (BFS) on its own associative diagram to iterate further outwards to other associated nodes:

Initial Prescription and Relief
      |
      |
Craving and Escalation
     /    \
    /      \
Withdrawal Symptoms   Impact on Life and Relationships
     |     \                /          |         \
     |      Physical Symptoms   Social Isolation  Strained Relationships 
     |      Emotional Symptoms   \         /         \      
     |                          Deceptive Behaviors   
Attempts to Quit            Attempts to Quit
     |                            |
     |                            |
Support Systems            Support Systems
     |          |           |             |           |
     Rehab   Counseling   Relapse   Family Support  Peer Support  Medical Treatment

And, in the spirit of infinite semiosis, we could continue onwards in this fashion ad infinitum.

Extended associative networks

Indeed, I have experimented with this approach. Starting with a core concept, and then asking the model to extend its associations outwards in a graph based representation. The largest graph I’ve generated has nearly 10k nodes. At the link you can access one node and traverse around the graph. I’ve also created other graphs, such as this one which started with the node “curiosity” and then branched outwards from there:

Zooming in on it, we can begin to get a sense of its structure:

We see dense clusters of associated words. Sometimes a single word with many 2nd order associations branching off of them:

These patterns can combine to create “peninsulas” of associated words, such as this one related to evolution and genetics:

Other times we see densely connected nodes with some long-distance connections, such as “Obscure” which connects to locally placed nodes like “Obfuscate,” “Clarify,” and “Shroud” while also connecting to more distant nodes like “Esoteric.” Where does esoteric connect?

“Esoteric” acts as a “bridge” between the nodes like “Obscure” and “Enigmatic” and the more distant nodes “Mystical” and “Occult.”

Occult itself has some local nodes, perhaps more obscure 2nd order nodes that are not connected to anything else, like “Gnosticism” and “Santeria” while also connecting to more connected nodes like “Supernatural.”

And so on. Already you can get a sense of the wide range of concepts, vague and specific, that are surfaced in the associative network. It suggests the presence of some kind of interlinked landscape of concepts whose topography and properties we can explore in the network.

Indeed, we could say that by creating such an associative network we are getting a map of the model’s “mental lexicon.” I’ve taken this lexicon approach further, and asked it to auto-generate definitions and descriptions of relations for each concept node in the graph. For example, the entry for “Gnosticism” has the following entry:

Gnosticism

Group: 9

Definitions

A religious movement embracing various ancient Greek philosophies and oriental mystical systems, characterized by belief in esoteric knowledge and dualism.
The thought and practice of various cults among certain sectarian groups in the early Christian church who claimed insight into esoteric truth.

Examples

Gnostic beliefs held that the material world was created by an evil or imperfect spirit, and that salvation could be achieved through gnosis, or esoteric knowledge.
The Gnostic Gospels, discovered in 1945, shed light on the beliefs and practices of early Gnostic sects.

Relations

associated with: [[Occult]] ← Gnosticism, an ancient religious movement emphasizing esoteric knowledge, is often associated with occult beliefs and practices.

Having such entries gives us access to the model’s possible representation or account of the concept, along with example uses it might use it in a sentence, along with a description of the kind of relation it has to other concepts. Having a multiplicity of relations might help us “triangulate” a concept in relation to other concepts. Consider for instance Occult’s multiple and lengthy entry:

“Occult” - Relations

associated with: [[Mysticism]] → Mysticism is often associated with occult practices and beliefs.
characteristic of: [[Esoteric]] → Occult knowledge and practices are often considered esoteric, or understood by only a select few.
involves: [[Supernatural]] → The occult deals with supernatural or paranormal phenomena that are beyond scientific understanding.
associated with: [[Paranormal]] → Paranormal activities and phenomena are often associated with the occult.
includes: [[Alchemy]] → Alchemy, the attempt to transmute base metals into gold, was considered an occult practice in medieval times.
includes: [[Astrology]] → Astrology, the study of celestial bodies and their influence on human affairs, is considered an occult science.
includes: [[Witchcraft]] → Witchcraft, the practice of magic and sorcery, is often associated with occult beliefs and practices.
includes: [[Divination]] → Divination, the practice of foretelling the future or gaining hidden knowledge through supernatural means, is an occult practice.
includes: [[Spiritualism]] → Spiritualism, the belief in communicating with the spirits of the dead, is considered an occult practice.
includes: [[Necromancy]] → Necromancy, the practice of communicating with the dead, is an occult practice often associated with black magic.
associated with: [[Theosophy]] → Theosophy, a philosophical system combining Eastern and Western mystical traditions, is often associated with occult beliefs and practices.
includes: [[Kabbalah]] → Kabbalah, the Jewish mystical tradition, is considered an occult practice by some.
associated with: [[Gnosticism]] → Gnosticism, an ancient religious movement emphasizing esoteric knowledge, is often associated with occult beliefs and practices.
includes: [[Hermeticism]] → Hermeticism, a philosophical and esoteric tradition based on the writings of Hermes Trismegistus, is considered an occult tradition.
includes: [[Rosicrucianism]] → Rosicrucianism, a secret society professing beliefs in esoteric wisdom and the occult, is considered an occult movement.
associated with: [[Wicca]] → Wicca, a modern pagan religion emphasizing magic and reverence for nature, is often associated with occult practices.
includes: [[Voodoo]] → Voodoo, a religious practice involving magic and the worship of deities and spirits, is considered an occult practice.
associated with: [[Santeria]] → Santeria, a syncretic religion involving the worship of deities and the use of magic, is often associated with occult practices.
is a: [[Sorcery]] ← Sorcery is considered part of the occult, which encompasses supernatural, mystical, or magical beliefs and practices.
category: [[Alchemy]] ← Alchemy was considered an occult practice, dealing with supernatural or mystical beliefs and practices.
is a part of: [[Witchcraft]] ← Witchcraft is considered part of the occult, which encompasses supernatural and mystical beliefs and practices.
involves: [[Confidential]] ← Confidential matters may involve occult or hidden knowledge.
is used in: [[Incantation]] ← Incantations are often used in occult practices and beliefs involving the supernatural.
characteristic: [[Obscure]] ← Obscure practices or beliefs can be considered occult or supernatural.
is associated with: [[Mystical]] ← Mystical beliefs and practices are often associated with the occult or supernatural.
is a type of: [[Necromancy]] ← Necromancy falls under the broader category of occult or supernatural practices.
encompasses: [[Magic]] ← The occult encompasses supernatural, mystical, or magical beliefs and practices.
is a part of: [[Wizardry]] ← Wizardry is considered part of the occult, involving supernatural or magical beliefs and practices.
is related to: [[Supernatural]] ← The occult deals with supernatural or mystical beliefs and practices.

Here we see a fairly comprehensive and diverse list of relations. We immediately notice the different kinds of relations used, from “is a part of” to “is associated with” to “includes”. Some relations are specific, like Wicca or Santeria (outbound relations), while others are even broader, like Magic or Supernatural (inbound relations). The list as a whole gives a sense of the model’s “umwelt” for this concept—the world of related concepts, broad and specific, that intersect with the core concept (in this case, “Occult”). It also starts to give us a picture of the kinds of knowledge the model has access to, from high level concepts like Magic to the more obscure traditions like Wicca, Rosicrucianism, or Santeria (none of which I knew about before doing this exploration). In this sense, we might say that by mining Claude’s latent associative graph, we are surfacing and exposing latent concepts within its inner structures and thus making them available to further explore.

This approach is deeply epistemic. I’m not sure that it has an application beyond creating elaborate knowledge graphs and lexicons, whose specificity might rival or even surpass existing encyclopedias and lexicons. This might be a useful tool to think with or to explore the local regions of a concept you’re thinking about, helping you scaffold what related knowledge exists in proximity to a concept. It could be used as a kind of research aid, or perhaps as a tool for getting a glimpse at the field of knowledge as it perhaps to a topic.

Consider for instance the node “Genome” and its locally related nodes:

We could hardly say it’s an extensive account of concepts related to the genome. However, it does surface high level related topics from DNA to Gene Therapy that one could use to help orient against the topic and explore further. I could imagine an interactive tool whereby one can progressively expand nodes to their related concepts, with the ability to delve more specifically or broadly, thus parameterizing the kind of associational expansion one does on a topic. Indeed some tools already exist which operate in this spirit, like the delve tool, which lets you do a branching traversal of related concepts.

In essence, this kind of approach is allowing you to traverse the model’s inner associative networks, thus moving you across the associative manifold, pin-pointing concepts in latent space, the proximate concepts, or distant relations. It invites us to think about what new interaction paradigms are possible, and what kinds of epistemic tools we could build to augment our ability to learn about concepts from latent space. Of course it’s worth cautioning about the potential for machine hallucination. The models provide us with a “fuzzy” picture of knowledge and its contours more so than a definitive account. And yet, knowing this, can the relational aspect of synthetic knowledge graphs provide a useful network for exploring ideas? What useful properties of such a network might help us understand how to build atop this paradigm?

From my own explorations in traversing the model’s latent structures, I’ve observed some nodes have a kind of “small world” property, where a concept might have relations to many densely connected local nodes and some fewer distantly located nodes. Other “central” nodes seem to have many long distance connections. Sometimes two nodes will share a “cluster” of more specific nodes, such as in the examples below:

This seems to highlight the specific way in which two concepts share commonalities, while also highlighting their differences (in the sub-nodes that are not shared). We might think of these shared and not shared 2nd order nodes as “features” of the 1st order node, showing how higher-level concepts have prototypes in more specific concepts.

Perhaps what’s most surprising is how diverse an associative network becomes when starting with just a single concept like “Curiosity.” In the network above we’ve seen concepts ranging from the occult to genetics, and the vast network that somehow connects the two. It gives us a sense that meaning is endlessly deferred and that semiosis indeed is infinite. How we can chart and explore this endless frontier is something I’ll continue to explore in future Substacks.

johan’s substack

Discussion about this post