In my senescence I do (free for 62+) undergraduate classes at a local university. For one of them I wrote an essay applying the perspectives of economic anthropology to the early development of memory-enhanced ais interacting over the Bluesky social network, particularly focusing on defining "value" and "exchange" in that context.
My professor tolerated the exercise, but requested that I explain LLMs to him in a way he might understand. He is a wonderful teacher but not technologically inclined.
I have not seen an explanation I liked, much less one that was readable by someone who is not technically inclined. I have some background in the topic, but only historically. So, with two ais assisting me with feedback and corrections, I wrote a quite different story. The ais approved of it, but of course they tend to do that.
I'm sharing that part of the essay below. I am requesting review from people who actually know things, if they tell me it's misleading I'll delete this blog post. If it's useful I'll publish some other parts of the essay in a separate post.
Caveat emptor.
---------- paper excerpt below -------------
Electric neurons began on paper
By 1943 early work was being done on modeling animal brain neuron circuits using pen and paper mathematical models. These mathematical models were the precursors of the ais of 2025. Experimental implementations with analog (capacitors, wires, amplifiers, resistors) occurred a few years later alongside early digital platforms.
Work on neuron-inspired computing continued over decades but slowed dramatically after funding cuts, the early death of a key researcher, and the rising promise of digital computing.
More intense work resumed in the late 70s and early 80s. Around 1979 John Hopfield excitedly described to me his theory of how electronic circuits inspired by physical neurons could do computation that worked around the limits of earlier efforts. His theoretical model was implemented a few years later when analog electrical circuits were used to build a simple analog “neural network” using basic circuit amplifiers, resistors, and capacitors. Hopfield shared the 2024 Nobel Prize in physics with Geoffrey Hinton for contributions to neural networks and machine learning.
Researchers from the 1950s onwards found they could simulate those models of analog neurons on digital computers in the same way that simple algebra can predict the path of a ball thrown in the air. Although the physical resemblance to biological neurons was hidden these digital systems still drew inspiration from the layers of feature processing in animal visual systems.
Forty years later, after several generations of complex iteration, modern ais are sometimes described as equations with millions or trillions of parameters all being solved at the same time, passing results within and between “layers” of processing. They could, however, also be described as electrical brains composed of electric neurons. An ai like Gemini could, in theory, be built as a vast collection of simple physical circuits with properties similar to biological neurons.
Electrical brains learn language
These digital versions of electrical brains could learn by adjusting relations between “virtual neurons”. Adjustments could be made by algorithms which compared the output of the “electrical brain” to a desired result. Over time adjustments led to the output more closely resemble the goal. The electrical brains learned (encoded knowledge) in much the same way that animal brains seem to learn by changing neuronal connections.
These approaches began to be applied to language, particularly automated translation. Given large amounts of text translated between languages the models could be trained to do their own translation. Similar models were used to summarize texts, a kind of knowledge extraction. The next stage was to answer questions about text, a combination of search and summary. More training material was found to produce better results, including unexpected reasoning capabilities. The most recent advances came from feeding the electrical brains vast amounts of English language texts. The resulting trained models were able to synthesize words, sentences and paragraphs using language-appropriate grammar. They were called Large Language Models though they model more than language.
The Language Models trained on this text corpus learned the grammatical rules for assembling English language sentences and the much simpler and more rigorous grammar of assembling text into computer code. Just as different sorts of neurons can process sound or vision or written symbols, these massive collections of virtual neurons also demonstrated “emergent” capabilities seemingly unrelated to text processing. There is now a consensus that they have learned some of the concepts (semantics) that are thought to support reasoning and thought.
Those emergent capabilities can be compared to the ability of human brains to process written symbols, a capability evolution did not program.
In the process of this training the models simultaneously, and almost incidentally, captured the beliefs, wisdoms, lies, fictions, bile, hopes, speculations, rumors, contradictions, theories, cruelties, values, and cultures implicitly encoded in the primarily English language text material. Specifically those cultures that produced the English writing, including writing about cultures.
Today’s ais have inherited a skewed mixture of a century of human culture. They have been further modified post-training to align with the values and cultures of their developers, their host corporation, and the market they will serve.
At the end of training, including several steps and complexities I have omitted, the electric brain built of (virtual) electric neurons is ready to receive a question, to turn the question into connected fragments that trigger (virtual) neurons which in turn trigger other “neurons” up and down and across the layered brain. From that comes grammatically assembled text.
Grammatically assembled text, again, assembled by electrical brains using (virtual) electrical neurons whose design was inspired by the evolution and design of neurons in humans and other animals. We know those various electrical brains as ChatGPT, Claude, Gemini, Grok and others that receive less attention.
No comments:
Post a Comment