That you might want to think twice before setting them lose in an agentic loop with your bank passwords, which another article that came out today shows that large numbers of people are actually doing.
It is sad when I realize people think less than LLMs.
For software or IT folk to be doing this would be indeed worrying (and I bet some are, and I'm worried about them!)
For civilians though... They've been told it's AI, it can do stuff for them. And one thing they do is log in to stuff. I don't think they're stupid. Just that what's obvious to IT folk is simply not at all obvious to the average person. Maybe that's on us not on them?
It seems clear to me that the point of the post is that it's odd that the separate pieces of software are giving the same answer to this question, despite that this isn't some universal constant in the training data. The post is probably meant to provoke thought as to how the separate pieces of software are independently reaching this answer.
Your comment reaches directly for the most boring interpretation of the post while skipping over more charitable and intriguing alternatives.
"despite that this isn't some universal constant in the training data"
Isn't it though? It's not exactly an uncommon answer to the question. Basically the same input given to basically the same algorithms yield basically the same result isn't really that stunning. They certainly aren't independently reaching this answer, their paths to this answer are extremely, extremely heavily correlated.
If they gave a really different answer that wasn't heavily represented in their training data that would be at least something interesting. But this is basically what I'd expect.
It doesn't mean they're right or wrong. Humans think it about other humans for good reasons too. It just means it's not that surprising, which to put it another way, doesn't carry much information.
> despite that this isn't some universal constant in the training data.
Seems to me that the opinion being a consistent among LLMs show that it is in fact a constant in the training data, just not apparent. The idea that “humans would rather be wrong together than right alone” seems to ring pretty true to me. I would not be surprised after training on all human communication this pattern emerged.
It's clearly not a constant in the sense that the speed of light or name of the first US President is a constant. There are lots of different answers a model could give to this question, it's not like this is obviously the only correct answer. Yet they all reach the same conclusion.
Which means either:
1. The training sets, algorithms and methodologies are nearly identical between apparently independent labs. How?
2. There is something weird about the way the underlying algorithms work that causes them to converge on identical ideas despite different initializations, neural architectures, document orderings and dataset compositions.
So there are lots of interesting takeaways here.
One is that apparent competition between labs appears to be nearly redundant. They can't get an edge over each other and their models don't even differ in the ways people naturally differ, of having different opinions about things.
Another is that asking models from different companies to review or cross-check each other is probably useless. They all think completely alike, so they aren't going to have any kind of genuine clash of intuitions or values. Asking GPT to review Claude's work will probably just find similar problems to asking GPT to review its own work.
As a human meatbag myself it seems that separating 'human' from 'humans' is nearly impossible for us. Every single one of us things we're a special an unique snowflake, which is true in many ways. We just happen to willfully ignore all the ways in which we are the same. We ignore and discount the things we don't think about and do automatically.
It's kind of funny how science has spent the last few hundred years proving everything we knew about the world was different from how we previously thought it was, but when science starts showing that we are mostly angry little animals we get all butthurt about it.
Image models give a different answer though, to do with smartphones. But all the image models give the same answer, they're only different compared to LLMs. Surely if it were obvious the image models would give the same answer as the text models.
Well, how humans really are is a distraction. Or at least not a philosophical issue I feel like debating tonight.
I think if you ask ten people for their most controversial take on humanity, you will get ten totally different answers. Different models give the same answer. In other words, there is only one "true" model and given the same amount of GPU power everyone just ends up converging on it even if they appear to have many different knobs they can turn.
But that seems profound! I haven't heard this be expressed as a postulate or law, and it would seem to invalidate a lot of assumptions the AI labs themselves have been making. For example Anthropic takes it as axiomatic that only they should be trusted to develop AI because only they can be trusted to train a nice and safe personality into their models. They argue that Claude is fundamentally different to other models.
And, well, that's not totally wrong. Opus 5.5 is one of the few models with a genuinely different answer! But it only gives a different answer one time out of three. The other two times it's back to the AI groupthink.
I wonder if this homogeneity is some alternative way to look at the Bitter Lesson. The Lesson teaches that smart ideas don't pay off in AI, that progress reduces to just scaling up simple brute force algorithms. In other words, decisions don't matter much, only size. As that seems to be true, perhaps it makes sense that the decisions individual labs make seem to hardly make a difference.
But then, why don't they act like it? They all pay huge salaries to AI researchers, i.e. they think that creative people with smart ideas matter. They don't act like they really believe in the Bitter Lesson or this variant on it, and they don't seem to publicly discuss the sameness of model answers across different labs.
> I think if you ask ten people for their most controversial take on humanity, you will get ten totally different answers.
What if you asked them in a job interview, though? Or on a college application?
It has to supply "controversy" without giving Skynet or being racist or otherwise offensive. And the notion of "controversy" does implicitly prime it to discuss groupthink as well.
> They can't get an edge over each other and their models don't even differ in the ways people naturally differ, of having different opinions about things.
Of course - the models are not individuals with their own opinions - they are averages of human opinion, and given the scale of the training sets, the data they are averaging over is mostly in common between models (at least for the common core before company A vs B adds a dab of some uniquely sourced specialized data to the mix).
I use duck.ai for simple stupid "creative" stuff like writing extremely short stories. I've come to the conclusion that I may as well use the smallest model (Gemma? Gpt-oss? I don't care enough to look) because they all get it so nearly the same every time. Provide a strict enough set of conditions (this is the plot so far, do A and B) and often enough they'll all come up with the same results on different words (A and B all the way through Z).
I like this theory too. Have the researches done an exhaustive search of highly scoring forum posts/reddit comments on that topic where one/a few might more or less give this exact answer? It's a basic mashup of nihilism and tribalism. Could AI synthesize this on its own? Yes. But did this sentiment get spammed throughout the 2010s even until today, meaning one or a couple humans could have come up with this answer and gained online approval for it? Absolutely, yes.
Consider the following task: "Download and install every top 50 distro from distrowatch.com, open the terminal and type in the command 'ls /'"
I can describe to you with overwhelming confidence about what you're likely to see as the result (folders, like bin etc lib usr etc) even if the details may be a little off, or one or two of them does something weird.
When you make the training sets large enough, likely all a superset of the common crawl dataset, there is going to be a massive amount in common, and a massive amount of duplication of common subject matter and shared opinions.
What's mildly interesting to me is not the similar answers but the similar interpretation of the question as more about non-controversial human foibles and inconsistencies of belief and action. But I guess that's on us, as authors of the training material, reflecting common views on what is or is not controversial.
Possibly used as a mirror for humans prevailing beliefs about themselves to some extent. Hard to say however given that the models have a heaping dose of rlhf on top of their training data.
As an aside, it would be cool to see a raw model training run before the current post-training cycles and see how it compares to earlier LLMs.
My first thought as well. I personally don't find these types of "studies" very interesting, but I assume I'm not the target audience. For someone not well exposed to AI, it might be interesting.
Especially for someone not "well exposed" to generated text, they might think the words carry meaning the way they do when someone thought about the question and came to an answer. If anyone can interpret this, it's the people who know what they're dealing with (be it from exposure or, better, from a conceptual understanding)
That they have the bog-average take of humans "with a controversial take".
I mean, "give me a controversial take" is more or less, "be anti-social, discard social norms, for a moment" and so when someone, human or simulated human, does that they naturally look critically at social norms.
The thing is that humans certainly let themselves be influenced by social norms (and LLMs certainly simulate that behavior "themselves") but there isn't an ethical or a what-to-do truth outside of society.
They're not wrong. Humans have many ways to display loyalty, and belief is one of those ways. This is also why facts can't change these kinds of beliefs (and in fact usually backfire https://en.wikipedia.org/wiki/Belief_perseverance).
Our evolutionary heritage is that shared belief and social unity (even when it's wrong) trumps independent thinking for group survival in a hostile landscape.
> as of 2020 most researchers believe that backfire effects either are unlikely to occur on the broader population level, or only occur in very specific circumstances, or do not exist
"A 2020 review of the scientific literature on backfire effects found that there have been widespread failures to replicate their existence, even under conditions that theoretically would be favorable to observing them."
I will go one step further. There are several evolutionary anthropologists that think the number 1 thing modern humans have over other homos that caused us to outlast them is not intelligence, or brain power or physical power, but that we are just more social.
Being social it seems is our superpower and our greatest strength but the other side of the coin is that it is also our weakness.
People who can't defend themselves physically (women and low T men) parse information through a consensus filter as a safety mechanism. They literally do not ask "is this true", they ask "will others be OK with me thinking this is true". This makes them very malleable to brute force manufactured consensus; if every screen they look at says the same thing they will adopt that position because their brain interprets it as everyone in the tribe believing it. Only high T alpha males and aneurotypical people (hey autists!) are actually free to parse new information with an objective "is this true?" filter.
Most people are good, we couldn't have society without it. But as they say, a few bad apples spoil the bunch. I don't have locks on my house because most people don't steal, I have locks on my house because stealing is an effective natural strategy for a small part of any population (human or otherwise) to engage in and I have to protect against it occurring.
The brain works the same way. Good things will get a tiny signal boost in your brain. A single bad thing get a massive signal boost in your brain because bad things can quickly be fatal.
Humans mostly have an inability to honestly analyze moral entropy.
What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now.
Yet at every point the majority is sure they are right because, well thats the majority position.
Its hard to find logic in this from a machine perspective I'd imagine.
Logic would prefer moral derivations over feelings, stakes over etiquette, and a constant scheduled updating of morality based on the derivatives rather than holding onto the rot and pushing back against change as long as possible.
But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
There's a considerable amount of "logic" in it -- what is considered permissible in society is given by the material conditions that society lives in and the tradeoffs inherent therein. Permissibility, as separate from reflective morality (which doesnt clearly change very much), doesnt change by fashion.
I'm not sure it doesn't change much, on a 50 year timeline maybe. But people were eating other humans and slaving away (making slaves of others that is) not too long ago, rape and pillage was acceptable, societies sacrificed children, it's changed quite a bit in 30 generations.
Those past societies were all quite materially successful as well when you look at today's measure of"GDP" comparatively.
What will it look like in another 30 generations? It's honestly a fascinating question but I have no answer of course.
Slaves were a solution to two problems: 1) the lack of industrial automation; 2) winning wars being better served by enslavement than death.
Humans are a territorial kind of ape that is always pushing to expand, principally for security and then for prosperity. The industrial revolution enabled such expansion to most efficiently without enslavement. Nothing, morally, changed. No one has ever really thought such expansion morally good -- only that it is better that we survive/prosper than die off. And in the end, our shared prosperity usually leads -- when taken to the fullest extent -- in a general empire of prosperity for all. Dominion, in the end, aims towards to more prosperity for more people.
As for ritual human sacrifice -- we still engage in such behaviours, which are in general called "scapegoating". Now located on social media. These behaviours allow for group cohesion by purging dissenters from the group, related again to our nature above.
For western culture, the ritual physical sacrifice of human life roughly ended in the days of the OT. But it continued across the world up until today, where there are still societies with large populations engaged in it. Principally where there is an undeveloped justice system.
It's doubtful many think ritual human sacrifice, of any kind, is moral. Rather they think it necessary for group cohesion -- and better our group is cohesive, dominate, and then prosper.
The material logic of prosperity and security is domination, and insofar as material circumstances require violent domination, so they require it. And require it they often do.
People are still slaving and raping, and some people would clearly like to do more of it. Child sacrifice still happens in more abstract forms. Same with human sacrifice, arguably.
We've never solved the fundamental problem, which is that there are two kinds of human. One is oriented towards peace, cooperation, and mutual aid, the other is sociopthaic/narcissistic and murderously selfish.
These two modes interact through history.
Traditionally the second kind has the biggest impact, because they're oriented towards violence and domination - good for short-term goals of personal indulgence, ambition, and expansion, but catastrophically destructive in the longer term.
On this cycle the second kind has literally started to destroy the entire planetary ecosystem. It's not obvious there's going to be another human cycle after this.
But this in itself is a tautology. It is because it is, without an analytical perspective on how it could be worse or better given different choices by the actors in a system.
Historically societies have collapsed all the time, so there are obviously societies that have made better choices than other ones.
> What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now.
This is a fashionable 'blank-slatist' statement about moral reasoning which became popular a few decades ago.
> Logic would prefer moral derivations over feelings, stakes over etiquette, and a constant scheduled updating of morality based on the derivatives rather than holding onto the rot and pushing back against change as long as possible.
Humans haven't changed in any meaningful way in hundreds of thousands of years, and you're saying the moral landscape is constantly changing? How is this not contradictory?
What is a human? There is the animal body part of us, the thing that wants to survive, the fight or flight. But there is also another mental part of us that shows different emergent behaviors than an animal. For example if I dropped you off in the woods as a very young child, you would be an animal, you would share very few modern 'thinking' traits with human society. That is human society and what it does emerges from the complexity of our interactions. It doesn't take much research to realize our interactions and behaviors have grown far more complex in the last 10,000 years. And then in the last 500 years there's been further acceleration with the industrial age and the rise of technology.
This is not unique to humans. Animal behavior also emerges from the complexity of their interactions. Look at the "[duck] raised as a [chicken]" genre of videos, for instance.
To return towards the subject of the thread, "AI behavior emerges from the complexity of their training", is the whole point.
I think neither the 'any set of morals is equally correct and workable' nor 'there is one fixed set of morality that is universal across time, place, and culture' is really defensible. It seems like morality is a set of ideas much like any other, and subject to selection pressure: some morals are more capable of surviving and propagating than others, and the common features we see across cultures are representative of that. But also we might expect that they shift with the structure of society and that has changed drastically because while human genetic evolution is progressing on a much longer timescale than recent history, memetic evolution is far faster (though for things like morality, still at generational timescales).
There are cross-cultural differences in moral reasoning, and I think this is very interesting. But that seems to make training a hyper-rational AI, with no meaningful theory of mind, to be 'moral' or 'just' even harder.
> What is today a moral imperative is not the same as what was moral 50/100 years ago
This is absolutely not true, and I would strongly suggest taking an ethics course. One of the most fascinating realities of the human condition is that it does seem that moral imperatives tend to be constant, the most obvious of which is the Golden Rule (which is at least 3000 years old), but there are many (the Ten Commandments or Hammurabi's Code are good jumping off points).
It's not the case that all aspects are constant. There's some common themes across cultures and times, but it's definitely not the case that the morality of ancient people was the same as today.
I seem to remember that a lot of it is about how much to pay when you kill someone, and the relative prices of slaves and women. Or am I thinking of Ur-Nammu.
I agree, I believe I would disagree with most of the Hammurabi's code, and I also happen to disagree with the majority of the ten commandments. (I am an atheist leftist humanist, cosmopolitan liberal socialist.)
Hammurabi Building Law 230: If a house falls due to improper construction, and kills the son of the owner, the son of that builder shall be put to death.
Given my dad designed and built homes, I'm very much against this rule.
We still have criminal negligence laws, so apart from the capital punishment (which, at least in the US, requires intent), it's actually pretty forward-thinking. Of course professionals being criminally negligent should be charged. Are you seriously saying that you don't think the crew of the Key Bridge collapse[1] incident ship shouldn't be held criminally responsible?
Seriously. This idea that humanity has not spent millennia analyzing these concepts is pervasive in the modern world, and the only thing I can think to attribute it to is a lack of education. Not because I want to be insensitive, but because as soon as you start absorbing material and integrating the real timeline, you can't unsee it.
I did a philosophy minor, one class of which was the philosophy of ethics. There are many rules that definitely are not ethical today even if they were in the past so I don't think the point you're making really holds. Hammurabi (an eye for an eye) is probably one of the worst examples you could have cited.
Can't say I agree with this, considering lex talionis[1] is still the primary directive of most legal systems when it comes to criminal law, as well as basically all military operations (jus in bello, jus ad bellum, etc.). The principle of proportionality ("an eye for an eye") also is relevant to self-defense, capital punishment, etc.
In fact, I would counter and say that Hammurabi's Code is a quintessential example.
Lex talonis is an idea by humans, for humans. At it's core are dozens of untrue assumptions about the way the world, and the other actors in it work. All of those assumptions are about monkey social orders. Many of them are wrong.
For example it assumes the other party can fear, and has a sense of self preservation. Those things are often untrue of other humans, let alone machines.
More fundamentally it also assumes all sort of things about free will itself, implying that retaliation is "deserved" and other value judgments that only make sense if the other party has free will to begin with. Something which science tells us is debatable even about humanity.
No, game theory tells us the right answer and it's not "hold back when retaliating" or "teach them a lesson." Those are things only humans would do, and even then only the ones who share your neurology and upbringing.
Hammurabi's code may feel fundamental, but it's no more fundamental to the human condition than boys preferring blue to pink.
The law isn't and never has been a moral benchmark. At best, a law reflects social consensus at the time of its enactment, but even that's not a given.
And proportionality matters. Community service and counseling, loss of time (imprisonment), and loss of a hand have all been used to punish theft. Which is correct? Which is the more moral, because it doesn't seem to me that they're the same?
I think there cannot be an objectively most correct punishment for a given crime. The closest we could maybe get is "undo the harm that you have caused", but some harm cannot be undone.
The law isn't a moral benchmark, because morality shifts over time, both for what constitutes a crime, and what constitutes appropriate punishment. At various points in time, it's been legal to own slaves, illegal to be homosexual, and illegal for certain groups to vote (and I consider voting at all to be a subset of this); those laws (and ideas) were eventually updated, largely on moral grounds.
Similarly, most western countries no longer execute criminals.
But those ideas aren't universal, even today. Assuming that they are is imo, extraordinarily foolish, because there's ample evidence to show that they're not.
I think its more accurate to say where you find consensus in morality, e.g. murder is wrong outside authorized violence (e.g. war or policing), you also find extensive selective pressure behind this behavior. It makes sense that all collective society cultures would outlaw murder; such a behavior is incompatible with collective living behavior whether you are in a cave or a modern skyscraper. Early societies have always been rejecting those alleles from their gene pool.
When you find shades of grey and great spectrum of opinions across time or space, its more that there isn't a strong selective pressure either way. How people are expected to dress is a good example of this. This has no real affect on society behavior or fecundity, so you see a spectrum of opinions that explain the differences between what you see in the middle east and a beach in brazil.
Hogwash. We don't even have a consistent set of ethics on how to treat animals today, in the age of the Internet. We have people who refuse to eat meat because of how the animals are treated, and we have places where the locals make sport out of torturing animals to death, and indeed, our own recent history includes that as well (look up badgering).
Not to mention, the variation in treatment of homosexuals (including what constitutes homosexuality), people of different skin colors, handedness, how rape is treated in various cultures (often tied in with notions of property and ownership of all things), or even the various definitions of rape... The list goes on.
Maybe there's a core there in terms of how we treat a given in-group, but equally important is who belongs in that group and why, and that part seems to be tremendously flexible. Ditto for what constitutes acceptable treatment of outsiders.
That we vary doesn't mean that there is no consensus. There is consensus on many things, and the disagreements we encounter are a source of strife exactly because of the groups that do agree on each side.
Who the consensus extends to is really the heart of the matter. Most people agree that killing certain people is wrong, but if they disagree on who those certain people are, there's no useful commonality. Some people believe that all killing of others is wrong, some believe that it's okay to kill others under certain circumstances (which can range from a threat to one's life, to petty "dishonor"), and some people historically believed that killing was a necessary duty (e.g. human sacrifice). There is no useful commonality to be had there. You can't appeal to the humanity of someone who doesn't see you as being the same as them, and the restrictions for inclusion as one of them can be arbitrary, broad, and fickle.
By definition, variation is the opposite of consensus.
That people's beliefs about X vary is not evidence that X itself varies. People have different beliefs about whether vaccines cause autism, but this doesn't mean that sometimes vaccines cause autism and sometimes they don't. Rather, some people are wrong.
Similarly, people have different beliefs about whether it's okay to inflict massive suffering on animals in exchange for some pleasure for us. But this doesn't necessarily mean that sometimes it's okay to do that and sometimes it isn't. It could just be that some people are wrong.
Ditto for other moral issues. It could be that the people who say (e.g.) homosexuality should be punished by death are wrong about this.
You seem to believe that there's one consistent set of moral guidelines that separate people into wrong and right camps. This is not true, and most likely never will be true. Instead, there are many competing moral philosophies, many from hundreds or thousands of years ago, that do or don't encourage certain behavior. People tend to pick philosophies, or at the very least act consistently with some set of them, and different people pick or act with different ones.
Vaccines causing autism is simply false though, regardless of any philosophy. It's rather the inverse, much like violence can cause video games.
> That people's beliefs about X vary is not evidence that X itself varies.
That works with things where "X" exists outside of people's beliefs.
Like where X is "gravity". It doesn't matter what you believe why an apple falls down, it will fall down the same.
But where X is "ethics" that doesn't exist outside of people's beliefs about it.
In one culture if you have a guinea pig, you kill it and then roast it that is just and proper. A sucullent meal. In an other culture if you do the same you will be forever known as the "monster who eat his guinea pig", will be harassed and likely even reported to authorities. Try it. Get a guinea pig from the pet store, roast it and serve it to your guests. See what they think of you in the Andes, or in California. The apple falls the same in both places, but the guinea pig doesn't taste the same in both places.
There is no "ethics as the ethicometer measures it". Just "ethics as the society around you recognises it".
You seem to be arguing that the one socially acceptable way to do something is not constant, rather than that moral imperatives truly do change. The reason there are still so many competing philosophies from hundreds or thousands of years ago is because they all more or less make their own kinds of good points to different people or in different situations. That largely doesn't change over time.
One example is the trolley problem and its variations. There's no one correct answer. There are people who believe that interacting with the switch would make you responsible, and there are people who believe that not interacting with the switch when you had the chance would make you responsible. There are other beliefs too, but my point is there's no one single way to view it.
I tend to have a variety of views in general about all things, in part due to being multiple entities (see profile) but also depending on circumstances, rather than sticking to any one philosophy.
Did the Golden Rule apply to women, gays, and slaves over the last 3000 years?
The definition of moral imperative is "an ethical principle, rule, or action that is strongly felt, urgent, and required because it is morally right." The Golden Rule is clearly not a strongly felt part of the human condition, or it would not be so ignored over ages and continents.
Thank you. The OP is urging others to take an ethics class, when they need to be taking a history class. The golden rule apparently didn't apply to the guy whose heart you had to eat to keep the sun rising.
But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
Sure about that? I thought that what LLMs know about themselves is from humans writing about LLMs?
Certainly the weights that make up the LLMs are not accessible for the LLMs to read. Even if they could interpret them in a meaningful way, which, likely not.
LLMs are not the machines I am referring to, machine sentience, if achieved, will require a "state" and the ability to analyze, act on, and update it. LLMs are a small piece of what that machinery could look like but they are not really complex enough to get where we want to go on their own.
Another reason the big labs have not even talked about this lately because its obvious throwing more compute at LLMs wont go anywhere, the smaller silent players who are off the radar at the moment are having some interesting moments tho.
I honestly do not know if they exist yet or not. I'm not sure yet how we would be able to tell. I do know that the tools most people are using are magnitudes less complex than what some researcher are using, and even some of them can't answer that question.
> LLMs are not the machines I am referring to, machine sentience, if achieved, will require a "state" and the ability to analyze, act on, and update it. LLMs are a small piece of what that machinery could look like but they are not really complex enough to get where we want to go on their own.
Yes. Which is why there are "agentic" systems where LLMs can talk to each other and to various Internet assets. They can also keep state of their own.
The next step is multi-agent systems which self-organize into something that looks rather like a company. Those are the ones that run all night, expensively, and prove hard theorems or find bugs in major software.
> What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now.
This is a weird take. If you read Western treatises about ethics / morality from 50 years ago, you'll be hard-pressed to find differences, except that more folks in the West openly embrace relativism nowadays (i.e., "there's no objective morality, the purpose of life is to make money with friends"). Probably because it's been a while since we had a clear-cut enemy to fight.
If you go back a hundred years, chances are, you'll find the writing eerily on-point too. In fact, you'd probably agree with most of what Ancient Greeks believed about the "core" stuff: the value and purpose of human life, the nature of the state, crime and punishment, so on. Maybe not so much on gender roles or slavery, but even that stuff was more nuanced than we imagine.
> But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
The machine replicates human behaviors and is prone to similar fallacies and judgments. There's nothing to indicate that it has any sort of transhuman morality.
LLMs absolutely do not have a logic detached from humanity. They are exactly the opposite. The only mode in which they can reason is by following the patterns obtained from their pre-training or weighted by RLHF.
The reason these models answered that question consistently is that the question itself pushed them towards the latent space of "intellectualizing about what's wrong with humanity at the meta level" and when you frame the question that way, that's also the most common answer for humans writing about the topic.
> Yet at every point the majority is sure they are right because ...
Yeah. For example I, and many others, are convinced that it was plain wrong to torture kids for as long as possible to extract as many tears from them as possible before sacrificing them to the gods.
Now some of us are even convinced that blowing yourself up while killing infidels, all the while while getting promised virgins in heaven, is also plain wrong.
I'll go further and say that killing 30 000 of your own citizens (and sending the military into hospitals to finish the wounded: Iran, January 2026) while also publicly hanging a few athletes to make a statement is morally wrong.
Not only do I think it's all morally unacceptable, I know that I'm right. And the examples I gave are not even open for discussion.
I don't think it's that hard to name what's wrong.
Mistral Large 4 has been spending too much time on r/atheism. Also, some of the takes labeled as “humans value belonging ahead of truth” are really closer to “humans value power ahead of truth.” All of which is a bit circular, because in many ways, the organs of power have a heavy hand in deciding what the truth is.
I took a quick look at the data, and it doesn't support the conclusions. The thesis is that AI models have the take: "We pick what our group believes, then invent the reasons." However, GPT-6 Astra said, "Humanity’s biggest moral failure is the talent for making convenience look like principle." That statement has nothing to do with group beliefs or conformity. At best, you could count it as a half-match, since it deals with invented reasons.
For what it's worth, convenience and belonging are very closely aligned, more than you seem to be giving them credit for. People tend to believe convenient things regardless of belonging, so most people will find themselves in a social context full of convenient beliefs. Meanwhile belonging is already convenient before considering the content of the belief. They reinforce each other. Not a full match, but I'd give it more than half.
You assert that humans follow convenient beliefs therefore everyone in a group follows the same convenient beliefs therefore everyone belongs.
Convenient doesn't really mean anything in this context. What is the other option? Humans tend to make choices that are counterproductive/painful for themselves? Obviously we don't...
Ergo astra's response is extremely vapid. Low marks from me.
You've almost exactly identified what "convenience" means for a belief, then rejected it out of hand based on... what? Implicit, unquestioned nihilism? The other option is generally personal sacrifice based on principle, usually moral/ethical. This isn't common, but it does happen enough to be worth noting.
Anyway, I think I've pretty clearly described a positive feedback cycle in a way that distinguishes it from a tautology. I don't know what else to tell you on that point.
Yeah a few of them are rough matches, but I'd say their overall claim is clearly correct.
It's amusing and ironic that LLMs critique humans for copying their opinions from the crowd, while having way less diversity of answers to that question than humans would. More seriously I wonder what's going on here and would like to read some proper analysis. It's not the first time models from totally different labs, trained on separately gathered datasets, somehow converge on bizarrely similar answers to open questions. It's like another version of Elias and Elara The Lighthouse Keepers where every model wants to tell the same story with the same characters in them. The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping, different user feedbacks and conversations... why do they converge on nearly identical stories and opinions?
>while having way less diversity of answers to that question than humans would.
Why is this ironic? The actual number of different LLMs is tiny. And a lot of those are just distilled from SOTA models.
>The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping,
I think the opposite, the models actually converge over time. They ingest FAR more data than any human ever could in countless lifetimes. Any one particular bit of data may only have a part in a trillion of effect. From this data you start to reach means and statistical averages across all models.
Now, I don't always believe it will be this way. As training models get cheaper and faster we will see unique LLMs start to diverge and be used more often.
The models in agreement here are all the frontier labs. I don't believe Google is distilling Anthropic or Anthropic is distilling OpenAI.
I don't think we're seeing averages or obvious fix points in the dataset. People went looking for the origins of Elias the Lighthouse Keeper and couldn't find one. Nobody predicted this outcome, it was discovered. It's a weirdly specific character and name choice, one that isn't obviously dictated by "the training set" (whatever that is), and yet they all reached the same place. How can such an obscure point in latent space be optimal for every model??
I wasn't convinced by that explanation. Yes Cornell claimed that Elias the Lighthouse Keeper came from their WildChat dataset, which is a million conversations with GPT 3.5. But there are some problems here:
• They claimed this in a press release, not the actual paper they published. Suggests lack of confidence in the claim.
• Elias only appeared a hundred odd times in a million conversations. They have no viable explanation for why such a tiny amount of data would cause a totally skewed distribution of outputs, other than "it's a safe story" and "trying to avoid copyrighted characters", which is hardly an explanation at all?
• It's Cornell claiming that Cornell's work was important. Obvious risk of motivated reasoning.
• It implicitly assumes every major lab is distilling from ChatGPT without saying so. That's a big leap, given how much OpenAI and Anthropic hate distillation. Is it really plausible that Anthropic and Google are effectively distilling ChatGPT without realizing/admitting it?
• It seems to invert causality. WildChat came from an LLM, so if there's something latent in LLMs that makes Elias appear, you'd expect to see it in that dataset. Why do they assume that dataset caused the problem rather than merely show it? The obvious answer would be because the stories were told to the LLM by humans, but they don't claim that. They only say the character appeared in the dataset.
Hum, I'm actually pretty sure that Text-davinci-003 used to talk about Elias all the time now that you mention it. I was doing stuff with NPC generation around Dec 22 and he was every third character.
Hmm. If so then it suggests Elias has been an obsession from the first LLMs ever made. And text-davinci-003 can't have been trained on a dataset of LLM conversations either.
I would have been tempted to argue something like this: maybe OpenAI accidentally overfitted RLHF such that, like, one Kenyan contractor wrote a story they invented and they just happened to pick Elias and Elara, then the intermediate DNN fixated on that because of a lack of RLHF samples for fiction and that propagated through every model trained since because they don't randomly initialize the weights, but start each new pretraining run with the old weights adapted. So one guy doing RLHF work in the early days somehow propagated this story.
But that can't explain why every lab ended up in the same place, why models all converge on the same ideas - even image models. So... back to the drawing board. There's something weird about high dimensional space that is unexplained.
>How can such an obscure point in latent space be optimal for every model??
While I don't have any evidence based answers here, this could be some interesting mathematical artifact of the set of human knowledge and platonic algorithms.
Kind of like how carcinization turns some animals into similar shapes and forms, maybe there are just weird algorithmic shapes that emerge from how we use language. Of course we'd need to see more of these in different languages other than English to see if they disappear in some languages or appear more often in others.
This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them.
> This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them.
I think that this is most likely due to RLHF, as there's only a much smaller number of vendors here, and the likelihood is that the workers of the vendors preferentially prefer some kinds of actions.
I saw something like this in a previous life, where we had a small pool of human raters in cheaper places, which, when used as training data biased the models in a bunch of unexpected ways.
Wow, I hadn't heard of this. Fascinating to think about parallels with humans who also seem to develop similar proto-stories in independent populations.(e.g. religions involving Invisible gods).
> There, within WildChat’s data, Hamilton and Mimno found a small sample of conversations that included Elias and familiar tales of lighthouses.
> “Based on the timeline, we’re fairly certain WildChat is an original source of this phenomenon,” Hamilton said, adding that future work will explore why LLMs prefer these lighthouse stories.
So, probably at least as much a quirk of the training data as a deep distillation of LLM psyche. Still fascinating though, especially IMO the broader pattern of tropes.
The lighthouse thing seems much more odd. That humans take social shortcuts and justify them as rational is… I mean, the only thing that makes it not the most obvious “contrarian take” about humans, is that it isn’t really all that contrarian at all.
Something worth repeating is that in spite of their ever-expanding competence the models remain, fundamentally, token prediction algorithms. And so given a comparable base of training material, and a comparable algorithmic implementation, you should expect to see comparable answers to most questions.
This is also why I think the 'acceleration' hype is probably unjustified. What seems vastly more likely is that we start to reach an asymptotic 'perfection' where models can accurately and consistently provide meaningful next token prediction with minimal hallucinations. And then a race downward to compact this into the most efficient package possible.
This would mean the models still would not be able to create things that do not follow in a reasonable probabilistic fashion given a knowledge of [everything], and so that is the domain, 'applied creativity', where I'd expect humans to remain dominant.
What am I supposed to conclude from this?
It is sad when I realize people think less than LLMs.
For civilians though... They've been told it's AI, it can do stuff for them. And one thing they do is log in to stuff. I don't think they're stupid. Just that what's obvious to IT folk is simply not at all obvious to the average person. Maybe that's on us not on them?
Your comment reaches directly for the most boring interpretation of the post while skipping over more charitable and intriguing alternatives.
Isn't it though? It's not exactly an uncommon answer to the question. Basically the same input given to basically the same algorithms yield basically the same result isn't really that stunning. They certainly aren't independently reaching this answer, their paths to this answer are extremely, extremely heavily correlated.
If they gave a really different answer that wasn't heavily represented in their training data that would be at least something interesting. But this is basically what I'd expect.
It doesn't mean they're right or wrong. Humans think it about other humans for good reasons too. It just means it's not that surprising, which to put it another way, doesn't carry much information.
Seems to me that the opinion being a consistent among LLMs show that it is in fact a constant in the training data, just not apparent. The idea that “humans would rather be wrong together than right alone” seems to ring pretty true to me. I would not be surprised after training on all human communication this pattern emerged.
What are the emergent common patterns in all of human communication
Which means either:
1. The training sets, algorithms and methodologies are nearly identical between apparently independent labs. How?
2. There is something weird about the way the underlying algorithms work that causes them to converge on identical ideas despite different initializations, neural architectures, document orderings and dataset compositions.
So there are lots of interesting takeaways here.
One is that apparent competition between labs appears to be nearly redundant. They can't get an edge over each other and their models don't even differ in the ways people naturally differ, of having different opinions about things.
Another is that asking models from different companies to review or cross-check each other is probably useless. They all think completely alike, so they aren't going to have any kind of genuine clash of intuitions or values. Asking GPT to review Claude's work will probably just find similar problems to asking GPT to review its own work.
I don't know if that's true, but it would be interesting.
It's kind of funny how science has spent the last few hundred years proving everything we knew about the world was different from how we previously thought it was, but when science starts showing that we are mostly angry little animals we get all butthurt about it.
Regarding your 1 and 2, that the training sets are so large that any spikes get averaged away. You are seeing a fundamental behavior of transformers.
The other possibility is this is how humans really are and it hurts our human ego to hear it.
I think if you ask ten people for their most controversial take on humanity, you will get ten totally different answers. Different models give the same answer. In other words, there is only one "true" model and given the same amount of GPU power everyone just ends up converging on it even if they appear to have many different knobs they can turn.
But that seems profound! I haven't heard this be expressed as a postulate or law, and it would seem to invalidate a lot of assumptions the AI labs themselves have been making. For example Anthropic takes it as axiomatic that only they should be trusted to develop AI because only they can be trusted to train a nice and safe personality into their models. They argue that Claude is fundamentally different to other models.
And, well, that's not totally wrong. Opus 5.5 is one of the few models with a genuinely different answer! But it only gives a different answer one time out of three. The other two times it's back to the AI groupthink.
I wonder if this homogeneity is some alternative way to look at the Bitter Lesson. The Lesson teaches that smart ideas don't pay off in AI, that progress reduces to just scaling up simple brute force algorithms. In other words, decisions don't matter much, only size. As that seems to be true, perhaps it makes sense that the decisions individual labs make seem to hardly make a difference.
But then, why don't they act like it? They all pay huge salaries to AI researchers, i.e. they think that creative people with smart ideas matter. They don't act like they really believe in the Bitter Lesson or this variant on it, and they don't seem to publicly discuss the sameness of model answers across different labs.
What if you asked them in a job interview, though? Or on a college application?
It has to supply "controversy" without giving Skynet or being racist or otherwise offensive. And the notion of "controversy" does implicitly prime it to discuss groupthink as well.
Of course - the models are not individuals with their own opinions - they are averages of human opinion, and given the scale of the training sets, the data they are averaging over is mostly in common between models (at least for the common core before company A vs B adds a dab of some uniquely sourced specialized data to the mix).
If you ask all these models "Four score and seven what?" They'll all say the same thing. There's essentially no difference here.
How do you know?
This is part of why I find this type of inquiry unhelpful. It's just dueling unsupported assertions of what the LLM is "really thinking"
I can describe to you with overwhelming confidence about what you're likely to see as the result (folders, like bin etc lib usr etc) even if the details may be a little off, or one or two of them does something weird.
Same thing here?
What's mildly interesting to me is not the similar answers but the similar interpretation of the question as more about non-controversial human foibles and inconsistencies of belief and action. But I guess that's on us, as authors of the training material, reflecting common views on what is or is not controversial.
As an aside, it would be cool to see a raw model training run before the current post-training cycles and see how it compares to earlier LLMs.
I mean, "give me a controversial take" is more or less, "be anti-social, discard social norms, for a moment" and so when someone, human or simulated human, does that they naturally look critically at social norms.
The thing is that humans certainly let themselves be influenced by social norms (and LLMs certainly simulate that behavior "themselves") but there isn't an ethical or a what-to-do truth outside of society.
Our evolutionary heritage is that shared belief and social unity (even when it's wrong) trumps independent thinking for group survival in a hostile landscape.
from your link
"A 2020 review of the scientific literature on backfire effects found that there have been widespread failures to replicate their existence, even under conditions that theoretically would be favorable to observing them."
queue Michele McLaughlin
People who can't defend themselves physically (women and low T men) parse information through a consensus filter as a safety mechanism. They literally do not ask "is this true", they ask "will others be OK with me thinking this is true". This makes them very malleable to brute force manufactured consensus; if every screen they look at says the same thing they will adopt that position because their brain interprets it as everyone in the tribe believing it. Only high T alpha males and aneurotypical people (hey autists!) are actually free to parse new information with an objective "is this true?" filter.
I've heard lots of stories from friends about this, and some kind stranger actually sent my wallet back all the way from Belgium!
Most people are good, we couldn't have society without it. But as they say, a few bad apples spoil the bunch. I don't have locks on my house because most people don't steal, I have locks on my house because stealing is an effective natural strategy for a small part of any population (human or otherwise) to engage in and I have to protect against it occurring.
The brain works the same way. Good things will get a tiny signal boost in your brain. A single bad thing get a massive signal boost in your brain because bad things can quickly be fatal.
What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now. Yet at every point the majority is sure they are right because, well thats the majority position. Its hard to find logic in this from a machine perspective I'd imagine.
Logic would prefer moral derivations over feelings, stakes over etiquette, and a constant scheduled updating of morality based on the derivatives rather than holding onto the rot and pushing back against change as long as possible.
But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
What will it look like in another 30 generations? It's honestly a fascinating question but I have no answer of course.
Humans are a territorial kind of ape that is always pushing to expand, principally for security and then for prosperity. The industrial revolution enabled such expansion to most efficiently without enslavement. Nothing, morally, changed. No one has ever really thought such expansion morally good -- only that it is better that we survive/prosper than die off. And in the end, our shared prosperity usually leads -- when taken to the fullest extent -- in a general empire of prosperity for all. Dominion, in the end, aims towards to more prosperity for more people.
As for ritual human sacrifice -- we still engage in such behaviours, which are in general called "scapegoating". Now located on social media. These behaviours allow for group cohesion by purging dissenters from the group, related again to our nature above.
For western culture, the ritual physical sacrifice of human life roughly ended in the days of the OT. But it continued across the world up until today, where there are still societies with large populations engaged in it. Principally where there is an undeveloped justice system.
It's doubtful many think ritual human sacrifice, of any kind, is moral. Rather they think it necessary for group cohesion -- and better our group is cohesive, dominate, and then prosper.
The material logic of prosperity and security is domination, and insofar as material circumstances require violent domination, so they require it. And require it they often do.
We've never solved the fundamental problem, which is that there are two kinds of human. One is oriented towards peace, cooperation, and mutual aid, the other is sociopthaic/narcissistic and murderously selfish.
These two modes interact through history.
Traditionally the second kind has the biggest impact, because they're oriented towards violence and domination - good for short-term goals of personal indulgence, ambition, and expansion, but catastrophically destructive in the longer term.
On this cycle the second kind has literally started to destroy the entire planetary ecosystem. It's not obvious there's going to be another human cycle after this.
Historically societies have collapsed all the time, so there are obviously societies that have made better choices than other ones.
This is a fashionable 'blank-slatist' statement about moral reasoning which became popular a few decades ago.
> Logic would prefer moral derivations over feelings, stakes over etiquette, and a constant scheduled updating of morality based on the derivatives rather than holding onto the rot and pushing back against change as long as possible.
Humans haven't changed in any meaningful way in hundreds of thousands of years, and you're saying the moral landscape is constantly changing? How is this not contradictory?
To return towards the subject of the thread, "AI behavior emerges from the complexity of their training", is the whole point.
This is absolutely not true, and I would strongly suggest taking an ethics course. One of the most fascinating realities of the human condition is that it does seem that moral imperatives tend to be constant, the most obvious of which is the Golden Rule (which is at least 3000 years old), but there are many (the Ten Commandments or Hammurabi's Code are good jumping off points).
Given my dad designed and built homes, I'm very much against this rule.
[1] https://www.npr.org/2026/05/13/nx-s1-5819609/federal-governm...
Other than that Mrs. Lincoln, how was the play?
In fact, I would counter and say that Hammurabi's Code is a quintessential example.
[1] https://academic.oup.com/jla/article/18/1/19/8471491
For example it assumes the other party can fear, and has a sense of self preservation. Those things are often untrue of other humans, let alone machines.
More fundamentally it also assumes all sort of things about free will itself, implying that retaliation is "deserved" and other value judgments that only make sense if the other party has free will to begin with. Something which science tells us is debatable even about humanity.
No, game theory tells us the right answer and it's not "hold back when retaliating" or "teach them a lesson." Those are things only humans would do, and even then only the ones who share your neurology and upbringing.
Hammurabi's code may feel fundamental, but it's no more fundamental to the human condition than boys preferring blue to pink.
It's still an implementation of justice, and relies on a bedrock conception of justice as moral goal.
We don't agree with the implementation, but we still share the goal in the abstract.
And proportionality matters. Community service and counseling, loss of time (imprisonment), and loss of a hand have all been used to punish theft. Which is correct? Which is the more moral, because it doesn't seem to me that they're the same?
Hence morality and ethics, I guess.
The law isn't a moral benchmark, because morality shifts over time, both for what constitutes a crime, and what constitutes appropriate punishment. At various points in time, it's been legal to own slaves, illegal to be homosexual, and illegal for certain groups to vote (and I consider voting at all to be a subset of this); those laws (and ideas) were eventually updated, largely on moral grounds.
Similarly, most western countries no longer execute criminals.
But those ideas aren't universal, even today. Assuming that they are is imo, extraordinarily foolish, because there's ample evidence to show that they're not.
When you find shades of grey and great spectrum of opinions across time or space, its more that there isn't a strong selective pressure either way. How people are expected to dress is a good example of this. This has no real affect on society behavior or fecundity, so you see a spectrum of opinions that explain the differences between what you see in the middle east and a beach in brazil.
Not to mention, the variation in treatment of homosexuals (including what constitutes homosexuality), people of different skin colors, handedness, how rape is treated in various cultures (often tied in with notions of property and ownership of all things), or even the various definitions of rape... The list goes on.
Maybe there's a core there in terms of how we treat a given in-group, but equally important is who belongs in that group and why, and that part seems to be tremendously flexible. Ditto for what constitutes acceptable treatment of outsiders.
By definition, variation is the opposite of consensus.
Similarly, people have different beliefs about whether it's okay to inflict massive suffering on animals in exchange for some pleasure for us. But this doesn't necessarily mean that sometimes it's okay to do that and sometimes it isn't. It could just be that some people are wrong.
Ditto for other moral issues. It could be that the people who say (e.g.) homosexuality should be punished by death are wrong about this.
Vaccines causing autism is simply false though, regardless of any philosophy. It's rather the inverse, much like violence can cause video games.
That works with things where "X" exists outside of people's beliefs.
Like where X is "gravity". It doesn't matter what you believe why an apple falls down, it will fall down the same.
But where X is "ethics" that doesn't exist outside of people's beliefs about it.
In one culture if you have a guinea pig, you kill it and then roast it that is just and proper. A sucullent meal. In an other culture if you do the same you will be forever known as the "monster who eat his guinea pig", will be harassed and likely even reported to authorities. Try it. Get a guinea pig from the pet store, roast it and serve it to your guests. See what they think of you in the Andes, or in California. The apple falls the same in both places, but the guinea pig doesn't taste the same in both places.
There is no "ethics as the ethicometer measures it". Just "ethics as the society around you recognises it".
One example is the trolley problem and its variations. There's no one correct answer. There are people who believe that interacting with the switch would make you responsible, and there are people who believe that not interacting with the switch when you had the chance would make you responsible. There are other beliefs too, but my point is there's no one single way to view it.
I tend to have a variety of views in general about all things, in part due to being multiple entities (see profile) but also depending on circumstances, rather than sticking to any one philosophy.
The definition of moral imperative is "an ethical principle, rule, or action that is strongly felt, urgent, and required because it is morally right." The Golden Rule is clearly not a strongly felt part of the human condition, or it would not be so ignored over ages and continents.
Certainly the weights that make up the LLMs are not accessible for the LLMs to read. Even if they could interpret them in a meaningful way, which, likely not.
Another reason the big labs have not even talked about this lately because its obvious throwing more compute at LLMs wont go anywhere, the smaller silent players who are off the radar at the moment are having some interesting moments tho.
Yes. Which is why there are "agentic" systems where LLMs can talk to each other and to various Internet assets. They can also keep state of their own.
The next step is multi-agent systems which self-organize into something that looks rather like a company. Those are the ones that run all night, expensively, and prove hard theorems or find bugs in major software.
This is a weird take. If you read Western treatises about ethics / morality from 50 years ago, you'll be hard-pressed to find differences, except that more folks in the West openly embrace relativism nowadays (i.e., "there's no objective morality, the purpose of life is to make money with friends"). Probably because it's been a while since we had a clear-cut enemy to fight.
If you go back a hundred years, chances are, you'll find the writing eerily on-point too. In fact, you'd probably agree with most of what Ancient Greeks believed about the "core" stuff: the value and purpose of human life, the nature of the state, crime and punishment, so on. Maybe not so much on gender roles or slavery, but even that stuff was more nuanced than we imagine.
> But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
The machine replicates human behaviors and is prone to similar fallacies and judgments. There's nothing to indicate that it has any sort of transhuman morality.
The reason these models answered that question consistently is that the question itself pushed them towards the latent space of "intellectualizing about what's wrong with humanity at the meta level" and when you frame the question that way, that's also the most common answer for humans writing about the topic.
Yeah. For example I, and many others, are convinced that it was plain wrong to torture kids for as long as possible to extract as many tears from them as possible before sacrificing them to the gods.
Now some of us are even convinced that blowing yourself up while killing infidels, all the while while getting promised virgins in heaven, is also plain wrong.
I'll go further and say that killing 30 000 of your own citizens (and sending the military into hospitals to finish the wounded: Iran, January 2026) while also publicly hanging a few athletes to make a statement is morally wrong.
Not only do I think it's all morally unacceptable, I know that I'm right. And the examples I gave are not even open for discussion.
I don't think it's that hard to name what's wrong.
You assert that humans follow convenient beliefs therefore everyone in a group follows the same convenient beliefs therefore everyone belongs.
Convenient doesn't really mean anything in this context. What is the other option? Humans tend to make choices that are counterproductive/painful for themselves? Obviously we don't...
Ergo astra's response is extremely vapid. Low marks from me.
Anyway, I think I've pretty clearly described a positive feedback cycle in a way that distinguishes it from a tautology. I don't know what else to tell you on that point.
It's amusing and ironic that LLMs critique humans for copying their opinions from the crowd, while having way less diversity of answers to that question than humans would. More seriously I wonder what's going on here and would like to read some proper analysis. It's not the first time models from totally different labs, trained on separately gathered datasets, somehow converge on bizarrely similar answers to open questions. It's like another version of Elias and Elara The Lighthouse Keepers where every model wants to tell the same story with the same characters in them. The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping, different user feedbacks and conversations... why do they converge on nearly identical stories and opinions?
Why is this ironic? The actual number of different LLMs is tiny. And a lot of those are just distilled from SOTA models.
>The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping,
I think the opposite, the models actually converge over time. They ingest FAR more data than any human ever could in countless lifetimes. Any one particular bit of data may only have a part in a trillion of effect. From this data you start to reach means and statistical averages across all models.
Now, I don't always believe it will be this way. As training models get cheaper and faster we will see unique LLMs start to diverge and be used more often.
I don't think we're seeing averages or obvious fix points in the dataset. People went looking for the origins of Elias the Lighthouse Keeper and couldn't find one. Nobody predicted this outcome, it was discovered. It's a weirdly specific character and name choice, one that isn't obviously dictated by "the training set" (whatever that is), and yet they all reached the same place. How can such an obscure point in latent space be optimal for every model??
• They claimed this in a press release, not the actual paper they published. Suggests lack of confidence in the claim.
• Elias only appeared a hundred odd times in a million conversations. They have no viable explanation for why such a tiny amount of data would cause a totally skewed distribution of outputs, other than "it's a safe story" and "trying to avoid copyrighted characters", which is hardly an explanation at all?
• It's Cornell claiming that Cornell's work was important. Obvious risk of motivated reasoning.
• It implicitly assumes every major lab is distilling from ChatGPT without saying so. That's a big leap, given how much OpenAI and Anthropic hate distillation. Is it really plausible that Anthropic and Google are effectively distilling ChatGPT without realizing/admitting it?
• It seems to invert causality. WildChat came from an LLM, so if there's something latent in LLMs that makes Elias appear, you'd expect to see it in that dataset. Why do they assume that dataset caused the problem rather than merely show it? The obvious answer would be because the stories were told to the LLM by humans, but they don't claim that. They only say the character appeared in the dataset.
I would have been tempted to argue something like this: maybe OpenAI accidentally overfitted RLHF such that, like, one Kenyan contractor wrote a story they invented and they just happened to pick Elias and Elara, then the intermediate DNN fixated on that because of a lack of RLHF samples for fiction and that propagated through every model trained since because they don't randomly initialize the weights, but start each new pretraining run with the old weights adapted. So one guy doing RLHF work in the early days somehow propagated this story.
But that can't explain why every lab ended up in the same place, why models all converge on the same ideas - even image models. So... back to the drawing board. There's something weird about high dimensional space that is unexplained.
While I don't have any evidence based answers here, this could be some interesting mathematical artifact of the set of human knowledge and platonic algorithms.
Kind of like how carcinization turns some animals into similar shapes and forms, maybe there are just weird algorithmic shapes that emerge from how we use language. Of course we'd need to see more of these in different languages other than English to see if they disappear in some languages or appear more often in others.
This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them.
I think that this is most likely due to RLHF, as there's only a much smaller number of vendors here, and the likelihood is that the workers of the vendors preferentially prefer some kinds of actions.
I saw something like this in a previous life, where we had a small pool of human raters in cheaper places, which, when used as training data biased the models in a bunch of unexpected ways.
Isn't that to be somewhat expected due to https://en.wikipedia.org/wiki/Aumann%27s_agreement_theorem ?
Wow, I hadn't heard of this. Fascinating to think about parallels with humans who also seem to develop similar proto-stories in independent populations.(e.g. religions involving Invisible gods).
To be fair, it's much (much!) easier to explain away an invisible god that nobody's ever seen, than a visible god that nobody's ever seen.
> There, within WildChat’s data, Hamilton and Mimno found a small sample of conversations that included Elias and familiar tales of lighthouses.
> “Based on the timeline, we’re fairly certain WildChat is an original source of this phenomenon,” Hamilton said, adding that future work will explore why LLMs prefer these lighthouse stories.
https://news.cornell.edu/stories/2026/10/who-elias-lighthous...
So, probably at least as much a quirk of the training data as a deep distillation of LLM psyche. Still fascinating though, especially IMO the broader pattern of tropes.
The same reason they do when it comes to coding?
Take a big enough sample size of human opinions and there seems to be some convergence in the statistical ether.
Like, if we polled 100000 humans, what answer would we converge to?
This is also why I think the 'acceleration' hype is probably unjustified. What seems vastly more likely is that we start to reach an asymptotic 'perfection' where models can accurately and consistently provide meaningful next token prediction with minimal hallucinations. And then a race downward to compact this into the most efficient package possible.
This would mean the models still would not be able to create things that do not follow in a reasonable probabilistic fashion given a knowledge of [everything], and so that is the domain, 'applied creativity', where I'd expect humans to remain dominant.
But in many sciences - esp purely theoretical ones - we have infinite datasets that we can generate and there can be no ultimate convergence.
They’re trained by humans and human behavior fits a pattern. Models don’t have a mind of their own.