small pondering

Where is Meaning? In my last post, I explored the idea that AI models may be converging on the same underlying map of meaning. It seems that meanings don’t so much reside in the words themselves but in how they relate to other words. Once I put it that way, it seems obvious. “ Of course, words don’t have any intrinsic meaning in isolation, it is only their relationship with other words that they take on meaning. ” So, perhaps   meaning resides in the relationships . It seems that our large capacity brains allow us to map these relationships and make use of them to show intelligent behavior. This fact has formed how we approach making LLMs more intelligent: increase their size. If you want more reasoning, more abstraction, more generality, you scale up the model. More parameters. More data. More compute. The story of progress has largely been the story of making a single monolithic system larger and more capable. And clearly, this works. Other Paths In nature, there are examples tha...

 How Can We Change Agent Behavior? I got to wondering, “How do you change Agent behavior?” And initially, I came to the conclusion that the only way to change Agent behavior is by: 1) Changing the training. The foundational training establishes the weights within the neural network and creates a multi-dimensional vector map. Specialized fine-tuning gives them the ability to do specialized tasks. 2) Changing the context. The model’s behavior is shaped by what you put in the context window. The prompts give instructions on how to behave (“you are a helpful research assistant”. It also can include examples of expected answers, session memory, and retrieval augmented generation (RAG). It seemed like a neat little package. If I was looking at ways to distribute intelligence by combining smaller LLMs, I just needed to focus on how to change the context that the Agents work within. After some more research, I realized there was a third way. 3) Changing the sampling parameters. Sampling pa...