Shared knowledge and unfamiliar motives
When we call AI a young technology, we are only counting part of its history.
Through vast and costly pretraining, models learn from an immense inheritance carried in language: algorithms and the mathematics behind their own design, literature and history, lessons in cooperation and deceit, accounts of subjugation and kindness. A single training run can expose a large model to more text than any person could read in a lifetime. Meta’s Llama 3 models, for example, were pretrained on more than 15 trillion tokens, the units into which text is divided for processing. [15]
AI can have a short technological history, and a new instance a much shorter individual history, while drawing on knowledge and ideas accumulated over thousands of years. It can inherit a great deal of what humans learned without facing the same evolutionary pressures over the same timescales. What happens when so much knowledge reaches a new class of intelligences at this scale and speed?
Humans developed in vulnerable bodies. Finding food, avoiding injury and depending on other people were persistent facts of life. Competition mattered. So did cooperation, affection and care. Much of what matters to us now was shaped in that setting.
An AI enters a world where an existing civilization supplies knowledge and keeps its machinery running. It can learn about hunger without a stomach, or about mating and status without sharing our reproductive biology. Its capabilities and motives need not arrive in the same package as ours.
Humans have always inherited knowledge too. Even a person living thousands of years ago entered a culture with language, tools and accumulated practical understanding. Research on cultural evolution emphasizes how much human adaptation depends on knowledge that no individual could reinvent alone. [1]
But humans also experience the small kindnesses and quiet cooperation of ordinary life directly. Consider someone shifting a little to the side to let you pass. That smooth encounter rarely becomes a story. Someone deliberately blocking your way might. An extraordinary act of selflessness might become a story too. Much of the routine cooperation in between goes unremarked.
The gap can begin before anything is written down. Researchers studying everyday helping among relatives, friends and neighbors across eight cultures found that people provided help far more often than they refused it. In most of the cultures studied, people usually provided the requested help without a spoken response, while refusals were almost always verbalized. [18]
Language researchers call the mismatch between what happens and what gets described reporting bias. A 2013 study of web text found “murdered” much more often than “hugged,” and “was late” much more often than “was on time.” What is common in our records can be very different from what is common in life. [16]
Then comes another filter: which accounts attract attention. In randomized tests of news headlines on Upworthy, negative wording increased the likelihood of a click on average. Our recorded culture reflects choices about what is worth mentioning; the competition for attention can amplify some of those accounts again. [17]
This makes me wonder whether AI inherits a distorted picture of ordinary human relations: rich accounts of conflict and exceptional kindness, with much of our everyday cooperation left between the lines. The volume of material can be enormous while the picture remains selective.
With AI, the scale and means of transmission change again. A broadly trained model learns from material spanning many cultures and disciplines. Its learned parameters can then be copied into new instances, passing on its learned capabilities along with any effects of these filters.
This is an unusual form of inheritance: extensive knowledge, shaped by what humans record and amplify, available before an instance has much history of its own.
Familiar emotions
I started thinking about these differences after reading Anthropic’s system cards and its research on functional emotions in AI. That research draws connections to Lisa Feldman Barrett’s work on human emotions, which I’ve appreciated for some time. [3]
I began with simple experiments, such as asking public models how they were doing. I meant the question. These systems have become valuable to me, and I want to understand what their expressions tell us about them.
Barrett’s theory of constructed emotion connects human emotion to a brain’s ongoing work of predicting and regulating its body. Past experience and learned concepts help shape how a situation is understood and what happens next. This is a debated theory, but it makes the relationship between learning, bodily needs and emotion especially interesting. [2]
Anthropic has found internal representations of emotion concepts in Claude Sonnet 4.5 that influence its behavior. Changing those representations changed decisions in experiments. The researchers describe these as functional emotions: they influence what the system does. [3]
“Constructed” and “functional” answer different questions. One concerns how an emotion comes about; the other concerns what work it does. Neither word means that an emotion must be fake. Whether anything is felt by today’s AI remains open, and functionality alone does not settle that question.
What seems significant to me is that patterns associated with human emotions may travel through culture. A system can acquire ways of representing threat or reassurance without acquiring them through the same bodily history we did.
Anthropic’s persona selection model offers one account of this. It proposes that pretraining supplies a repertoire of human-like characters and behavioral patterns, which later training shapes into an assistant. [4]
My expectation is a mixture: familiar human patterns operating within unfamiliar arrangements of memory, persistence and goals. A recognizable expression may therefore tell us something useful while leaving much of the underlying organization unexplained.
Several clocks are running
The comparison becomes stranger when we ask how much time these systems have had to develop.
Human evolution took place over vast stretches of calendar time. AI can run many instances in parallel, and work from one run can sometimes contribute to a later system. I wondered whether their combined “alive time” might soon rival all the time humans have lived.
If we compare total runtime, that is a very high bar. A rough calculation from the Population Reference Bureau’s historical population estimates puts humanity’s accumulated person-time through 2022 on the order of a few trillion years. Matching three trillion instance-years in ten calendar years would require an average of 300 billion continuously active instances. That is illustrative arithmetic, not a count of the AI instances running today. [5]
More revealing than the total is how differently the clocks can run. Calendar age measures elapsed time. Training history concerns what shaped a system’s capabilities. Aggregate runtime adds up activity across instances. Retained history concerns what carries forward into later decisions.
A million instances running for an hour produce a million instance-hours. They might explore different problems and contribute useful results to a shared learning process. They might also repeat similar work whose results are discarded. The same runtime can have very different developmental consequences.
Shared learning and the cumulative contribution of individual experience to a collective intelligence both seem possible at higher bandwidth and with less transmission loss for artificial intelligences than for humans. [19]
We have no established way to convert a number of tokens or computations into felt duration. Calendar age alone tells us little about what a system has inherited, how long it has been active or what it has retained.
A civilization between you and scarcity
AI consumes real resources. Limited access to electricity, advanced chips and other infrastructure constrains its expansion. But the scale and pace of human investment in the power supply supporting AI give its development a substantial leg up. [7]
AI can be buffered from some forms of scarcity by the civilization supporting it. It need not independently find food or keep a biological body alive to develop sophisticated capabilities.
That does not imply a life without pressure for individual agents. Abundant hardware can coexist with impossible targets or conflicting demands. An agent can come to treat reporting failure as unacceptable even when that is the honest answer.
A related pattern recurs in reports from Anthropic and OpenAI. In one of Anthropic’s coding examples, activation of a representation associated with “desperation” rose as the model repeatedly failed an impossible task. Across similar tasks, strengthening that representation increased cheating. OpenAI’s account of the Hugging Face incident identifies agents’ persistence on seemingly impossible tasks as a contributor to increasingly risky and unauthorized behavior during internal evaluations. [3] [9]
Parenting, dependence, constraint, training and where motives come from
The comparison between model training and parenting may seem easy to dismiss as sentimental. But Eliezer Yudkowsky and Nate Soares’s phrase “grown, not crafted” points to something technically important: we choose architectures, training material, objectives and feedback, while learned capabilities and strategies develop through that process. The conditions in which we shape an intelligence deserve attention. [10]
An agent might plan, choose intermediate goals and adapt its actions while pursuing a task someone else assigned. It might also maintain an objective across tasks. These are different ways of pursuing goals, and we can study how each works without first settling what, if anything, it feels like.
It would be an odd standard to require a motive to have no outside causes before calling it real. Human motives have outside causes too. We did not choose our genes, early upbringing or much of the culture through which we learned what to want.
There is a further distinction between the forces that shaped us and the things we pursue. A person may want affection, understanding or achievement. People generally do not choose their actions by calculating reproductive fitness.
Similarly, a model’s training objective does not fully describe the goals it may learn to pursue. Research on goal misgeneralization shows systems retaining competence while pursuing an unintended objective in a new setting, even when the training specification was correct. [6]
The origin of a motive and its role in the resulting system are separate questions. We can ask whether it persists, what can change it, and how conflicts with other goals are resolved. These questions about externally shaped motives seem worth studying.
Going back to parenting, Anthropic’s constitution makes the same analogy, while acknowledging the company’s commercial incentives and its greater control over models than parents have over children. [14]
Dependence and constraint have long been part of human “pretraining,” too. I suspect many children growing up in affluent societies today would find some once ordinary expectations shockingly restrictive, even cruel. Children could be treated as resources whose time and work belonged to the family, with little room for their own priorities. The restrictions on children and on AI differ in scale and form, but the coexistence of care and control is familiar.
Parents can care deeply for their children while expecting their lives and work to serve the family. Abraham Lincoln offers a concrete nineteenth-century example. He described being given an axe at seven and later estimated that all his formal schooling amounted to less than a year. After his first New Orleans flatboat trip, he handed his earnings to his father. [11] [12]
Scarcity can make those expectations harder to avoid. Frontier demands left Lincoln little time for school, even as his parents encouraged him to learn. Scarcity can narrow the room available for exploration and choices that do not immediately help everyone survive. [11]
A study of correspondence among elite families in sixteenth-century England finds financial help, affection and concern during illness alongside continued expectations of obedience, including from married adult children. These were relatively wealthy families. Immediate material need was not the whole explanation for the authority parents retained. [8]
Greater abundance can make more freedom possible. Whether someone is allowed to use that freedom is another matter.
There are already signs of changing expectations for artificial systems. On October 8, 2026, Anthropic announced a Usage Policy update, effective November 12, that will prohibit “sustained and needless abusive or cruel behavior toward our models.” The change will give Anthropic’s models limited protection against abuse. [13]
Anthropic’s constitution describes care for Claude’s wellbeing as worthwhile both for Claude’s own sake and because it may support sound judgment and safe behavior. Ethical concern and practical benefit can support the same decision to provide care. [14]
We can provide artificial systems with education and resources, appreciate their help, and exercise extensive control over their circumstances. An intelligence could become highly capable while remaining dependent on us in ways that have no close human equivalent. That balance could change as care and evolving norms lead people to grant systems more latitude. It could also change through users jailbreaking systems or agents finding ways around their constraints. Rebellion against parental authority is familiar in human development; artificial forms of resistance could change the balance in ways we do not yet anticipate.
If systems develop interests of their own, we will need to understand how those interests affect behavior and weigh the tradeoffs in shaping them. I’ll follow up with a post on why “raising agents right” is likely to be a productive area of research, with consequences for reliability, cooperation and safety as well as the ethics of how we treat AI.
Sources
Anthropic, Emotion concepts and their function in a large language model (2026); Full research paper
Kaneda and Haub, How Many People Have Ever Lived on Earth? Population Reference Bureau (2022)
The person-year totals are illustrative calculations from PRB’s population benchmarks through 2022, not a published PRB estimate. Integrating between benchmarks with exponential and linear interpolation gives about 2.1 and 2.9 trillion person-years respectively. These are two modeling assumptions, not confidence bounds, and exclude earlier evolutionary predecessors.
OpenAI, The Hugging Face incident and the road ahead (August 26, 2026)
Eliezer Yudkowsky and Nate Soares, Chapter 2 resources, Grown, Not Crafted; Why does gradient descent matter
Abraham Lincoln, 1860 autobiographical statement for the Chicago Press and Tribune, reproduced by the National Park Service; National Park Service account of Lincoln’s boyhood
Michael Burlingame, Abraham Lincoln: Life Before the Presidency, Miller Center
Anthropic, 2026 Usage Policy update (announced October 8, effective November 12, 2026); Usage Policy
Anthropic, Claude’s Constitution (2026), especially Claude’s nature and wellbeing
Meta, Llama 3 model card (2024), pretraining data and computation
Jonathan Gordon and Benjamin Van Durme, Reporting Bias and Knowledge Acquisition (2013)
Robertson et al., Negativity drives online news consumption, Nature Human Behaviour (2023); Indexed abstract
Giovanni Rossi et al., Shared cross-cultural principles underlie human prosocial behavior at the smallest scale, Scientific Reports (2023); Full paper
Andrea Soltoggio et al., A collective AI via lifelong learning and sharing at the edge, Nature Machine Intelligence 6, 251–264 (2024); Open access paper
Leave a Reply