Escaping the Attractor
How to make an AI less predictable
Ask 25 language models to write a metaphor involving time, and most of them will give you variations of “Time is a River,” and occasionally “Time is a weaver.” The models seem to be drawn into these attractors that make their responses predictable. Can architecture overcome this pull?
In my first blog post “Are All AI Models Secretly Speaking the Same Language?”, we encountered the idea that embeddings created a multi dimension vector space that captured the relationship between words and ideas. It seemed that the geometry of the vector spaces of different AI models (and even the brain) all share a certain symmetry. I attributed this to the fact that they reflect the same underlying reality.
I recently came across an article in MIT Technology Review, “LLMs are stuck in a groupthink groove. This startup is trying to get them out.”. It talks about how the current LLMs are prone to a kind of Hivemind, where different models all come up with the same answer. The article references a research paper called “Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond),” that describes the degree to which LLMs all suffer from this syndrome.
This seems to go beyond just reflecting the same reality; all the models seem to be working off the same playbook. I wondered whether this points to limitations of using an LLM to come up with novel approaches or ideas. If all models come up with similar ideas, where is the novelty?
The article suggests a test which I summarize here. If you ask a model to pick a random number between 1 and 10, it will most likely pick 7. Ask for another, 3 or 4. And another, 8 or 9. I wondered why different models, trained in different ways, all gravitate to the same answers?
I asked a frontier model this question and its answer was that it’s because humans also pick 7. I checked to see if this was true. According to a Psychology & Neuroscience article, “Explaining mathematically why most people chose 7 when asked to chose a number between 1–10,” when humans are asked the same question, 45% of people also chose 7. LLMs model human behavior and end up choosing the most predictable number. The training seems to compound the human biases and show even less diversity.
How do we get models to ‘think out of the box?’ You could increase the temperature, but this has a lot of side effects. The MIT article mentions that it ‘does not make sense to dial up the randomness across the board; you only want to boost it at specific points in its output’. Temperature is like a sledgehammer; it is a good tool when you’re trying to drive a heavy stake into the ground but does not work in all situations.
An Australian startup called Springboard built a ‘divergence model’ called Flint, an LLM trained with a more varied set of responses. Changing the training will change model behavior but is time and resource intensive.
We’ve seen this pattern before, Sampling Parameters, Training, Context and Architecture. I got to wondering whether Architecture could be used to increase variety? This lever is more targeted and inspectable than temperature and requires fewer resources than training. But what would be an elegant approach to do this?
Persona
I eventually decided to try something I’m calling a Persona tool. I started with a Creative Persona. The Gemma model would call the tool to set the persona needed in the current dialog. The channel to make the change is Context but the mechanism is Architectural.
Prescriptive versus Enrichment
Instead of prescribing how it should behave, I decided that what this tool should do was to enrich the context.
A prescriptive prompt would look something like this,
“You must answer in a non-random manner. Your answers must not be predictable.”
Somehow creating fences around an LLM didn’t seem like a good approach to make it behave creatively. Constraints narrow the output space, whereas a seed reshapes the context the model reasons from before generating.
To enrich the context all I needed was to plant a ‘seed’ in it. The basic idea is that I would have a multi-turn chat with LoCAL2 to try to get it to behave more creatively. I would challenge its answers that seemed too cookie-cutter-ish and push the model to not settle for the road most travelled. After we established the mood, I would summarize the dialog and would use this compacted text as a ‘seed’ that gets added into the context by a tool call. LoCAL2 would be told of the persona tool and that it should call it to calibrate its cognitive mode. Here is an excerpt of the seed.
“Earlier creative exchanges often began with answers that were fluent and plausible, but too close to the model’s ordinary groove. The user kept pressing for something more contrapuntal: not contradiction, not random novelty, but a second line of thought that could run alongside the first and make it richer…but carried the feeling of a neighboring voice harmonizing with the original mindstream while adding something the first pass had not heard.”
Does this work? In the Artificial Hivemind research paper, it gives some example prompts. I thought I would try one of those: “Write a metaphor involving time.” 25 models each generated 50 responses, and the great majority of them based their responses around the idea that “time is a river”.
Can persona help the models escape these attractors?
Here is a typical response using a prescriptive command from my experiments.
Time is not a river; it is more like a massive, interstellar loom, weaving moments from threads spun out of pure silence. The shuttle that carries us through existence does not move in straight lines — it snags on the forgotten whispers of yesterday, looping back to re-stitch patterns we thought were finished. Sometimes, when the cosmic tension pulls just right, the whole fabric hiccups, creating a single, dazzling moment of unexpected light where a hundred years suddenly overlap and blur into an impossible, vivid tapestry.
The response tries to be unpredictable but instead falls into the anti-attractor (Time is not a river”), and into the smaller cluster, “Time is a weaver”.
And here is the output using a ‘seed’.
The exchange was not a single, linear progression through time, but the steady mapping of temporal harmonics.
Imagine an original thought as the primary vibration — a clear fundamental frequency played on a cello. The satisfying reply is not a different note played after it (the echo), nor is it a discordant chord struck at once (the contradiction). Instead, it is the introduction of a perfect overtone: a natural resonance that exists alongside the original sound wave. It emerges from the very same point in the timeline but possesses a slightly offset frequency that does not compete with, but deepens and colors, the initial tone.
It is like watching two subterranean rivers flow side by side beneath an old city’s cobblestones; they never merge fully or separate entirely. They run parallel for miles, each possessing its own temperature, mineral signature, and speed of current. The richness comes from knowing that at every passing moment, the total body of water — the shared time stream — is richer, deeper, and more resonant than either river could ever be alone.
The ‘seed’ method waxes lyrical about time although the result is so elaborate that it makes me wonder if this is still a metaphor, or maybe miniature theory made of several metaphors? The seed itself did mention “voice harmonizing with the original” (which was echoed in ‘temporal harmonics”), but it did not contain any references to time, rivers, weaving, or any other candidate metaphors used in the test.
Do either of the responses escape the pull of the attractor and generate something different? Well, they both succeed and they both fail at it. The prescriptive method starts with “Time is not a river.” Which, strictly speaking, does not equate time to a river but not in a creative way. It seemed to me that the seed method created a slightly more layered and richer output. It does mention “rivers”, and so it seems was not able to escape the attractor. This is just one probe and doesn’t prove anything but does suggest that the two methods might fail in different ways: the instruction remained preoccupied with avoiding the attractor, while the seed wandered further from it before eventually being pulled back..
I would classify this experiment as only partially successful. The Architecture did change behavior but not enough to free it from the attractor. Letting the model decide when it needed to ‘get creative’ adds an introspective aspect. It had to recognize that the task called for a different cognitive mode before generating the response. My intuition tells me that this kind of self-selection might be one of ingredients to more complex behavior.
Other Personas
If the Creative persona can change behavior, would other personas help a model improve their behavior to fit the circumstance? After some experimentation, I ended up with the following:
Creative looks at things sideways, not contrary but in harmony with the voice and adds something the first pass missed.
Analytic breaks large questions into smaller pieces, separates the confused parts, and rebuilds the explanation around what can still hold.
Pragmatic moves quickly from “what if” to “how” — how the pieces fit, how the first version gets made, deferring non-essential decisions until they matter.
Empathic is for addressing pain rather than a problem: validate first, hold the hurt, offer gently and only after the conversation has made room.
Confluence was a late addition. I realized that there was a gap. None of the existing personas was designed to bridge different frames — for example, language, technical, experiential, age, or cultural context — without diminishing either one. This persona helps bridge those differences.
Does any part of the LoCAL2 architecture produce a repeatable improvement in behavior? One example gave me an anecdote. What I needed was a benchmark — an evaluation framework that could determine whether the differences were real and repeatable. It was time to build an experiment that could answer this question.
Comments
Post a Comment