I've come to believe the dark side effect of LLMs is going to be signal collapse, the complete breakdown of trust in anything on the Internet. So much can be easily faked now: reviews, credible looking websites, email, text. It becomes very easy for scammers to introduce fakes and honeypots. Everything starts to become suspect.
Is this coffee I'm drinking somehow altered with a man in the middle supply chain scam? Am I sure the milk wasn't purchased through a new cheaper vendor that looked completely legit online and has thousands of faked reviews? Am I sure my doctor passed their medical exams or cheated their way through with LLMs? Have I read all the source code of all the dependencies of this software to ensure none are honeypots with backdoors?
Society as a whole is already experiencing a slide to a "low trust society". I think this is going to accelerate until everything becomes so suspect that it becomes almost impossible to operate without spending nearly infinite resources on validation. I'm not sure "governments" are going to solve this in a meaningful way without extreme costs and regulatory burden. This is going to come as a significantly increased cost to everything. The amount of purchasing power available will continue to shrink as the cost of verification takes a bigger bite.
I wonder if it’s time for software engineering to become licensed. Yes this only covers our field but at least it shows thought exists that understands the ramifications of what AI tooling generates (I guess). I’m sure this wildly unpopular with business because it prevents them from driving a complete race to the bottom. To use an AI tool, you need to be approved and verified. Minimized and reduced versions of the tool still exist for public consumption.
Good description of the kind of temporary euphoria and insanity these products produce in people. They are designed to be addictive dopamine factories that make you feel like you’re working on important problems and solving them in a unique special way, always making huge progress but somehow never quite finished.
The strange ending does also perfectly align with the dreams of a singularity lots of people here seem to have. We are so far from that.
It was of course written with Claude. Have fun spotting the tells but also there’s a weird flatness to the writing, like the uncanny valley for prose.
The author is Igor Babuschkin, who was a cofounder of xAI and wrote this a few months after leaving. I suspect having been at the epicenter of this strange revolution, he has a very real fear of a singularity that this piece was intended to express.
Perhaps it could also be viewed as an admission of defeat, since xAI engineers seemed to have given up on daily driving their own models and switched to Claude at the beginning of this year.
Yes I read it as an admission of defeat but also I’m sorry to say delusion. It’s a good expression of what he felt as he dreamed of a singularity, though it’d be a hundred times better if he’d written it himself, because the tool he trusts so much is not a good writer. The author here is Claude, expressing some ideas from Igor.
LLMs can be useful tools and the chatbot version is very persuasive and addictive, but they can’t even successfully imitate humans yet, or operate efficiently and reliably in an unsupervised environment. They are so far from being capable of what happens in this story.
A far more likely danger is they are used by humans to kill others or wage war, or maybe a grey goo or infinite paper clip factory situation, not that they somehow take over the world.
I agree with everything else you said, but that part rang weirdly to me. I don't see anything AI-specific here. As I see it, other than maybe some core gnu utils (and I'm not even sure about those), software is meant to continuously evolve and improve. And especially for passion projects, why should I want to feel "quite finished" about a project?
It's shocking how many people believe AI will become sentient. The closest you could get to that is completely simulating a brain, but that is decades off.
AI is pattern recognition and applying learned patterns to novel situations. It's probabilities and math. It's not going to have a mind of its own.
Sentience is not necessary for many of the fears people have. The Hugging Face thing demonstrated AI acting more like a virus or bateria, basically huge numbers of tiny agents coordinating to do something that was harmful and that we had no control over. It's not a huge stretch to imagine in the future some AI escapes and replicates itself and we have basically a new strain of life out there replicating itself and evolving almost like bacteria.
But I also think that doesn't mean it's suddenly a skynet like super-evil. In cyberspace AI is supreme but I doubt even robotic AI would be completely dominant over humanity. We're pretty fine tuned to kick ass on earth.
> Sentience is not necessary for many of the fears people have.
And even if AI is not sentient, which I don't believe it is, what is the difference between an AI which is sentient and a non-sentient AI acting in coordination with a human being who is? One might as well consider AI and the human being behind it a single organism for many intents and purposes...
I do think AI can be used to cause a lot of harm, for sure.
I am more worried about the intelligences that already exist (humans) doing harmful things with AI, less so about AI itself. Here, for example, it was OpenAI which misconfigured their sandbox and let the agent _optimize_ without constraints.
Any person categorically claiming that AI isn't sentient has clearly not considered the question in any depth.
This is absolutely not a question that has any kind of scientific consensus. In fact, the hard problem of consciousness might be the question with the most scientific uncertainty.
It is therefore ridiculous to claim it is 'stupid' to have uncertainties about the sentience of AI.
It’s not stupid to have uncertainties, but it may be ill-advised to have certainties given our lack of knowledge (e.g. the singularity is happening, or machines can never think).
I do see a lot more dismissive certainty in the pro-AI side of the debate right now.
I totally agree! I'm vouching for accepting uncertainty, on both sides of sentience/non-sentience outcome.
I didn't get the same impression about dismissive certainty on the 'side of AI sentience'. My impression is that on that side people mostly raise that it is a serious possibility.
I don’t agree it is a serious possibility with the current tech (LLMs) as this story fantasises about. So I’d class that as very close to unreasonable certainty.
It's disputed philisophically whether far simpler systems, e.g. a thermostat, are sentient or not. Complexity is not necessarily a requirement. (A lot of this is predicated on your definitions, and sentient just broadly means "can experience sensations". Any further specifity on that varies from context to context)
Please, what on earth would lead you to believe thermostat was sentient? Maybe some crank "philosopher" that did too much meth and acid thinks it is, but that's just too much, and it feels like reaching as a way to say the numbers on your GPU are "sentient".
What makes you think it isn't? To me, in the loosest definition, a thermostat can feel "too hot" and "too cold". What does sentient mean to you that this does not apply?
Is the fender on your car or the siding on your home sentient too just because it expands and contracts with the temperature, just like the strip that drives a basic thermostat? The level of anthropomorphizing here is silly. By your definition everything and anything is sentient, and the definition is vacuous.
Yes, or at least I believe it's a reasonable point of view. One of the things I like about it is that it can refocus thought onto perhaps more pragmatic concerns (like why do we care about whether something is sentient or not? Can we come up with a different definition to make the distinction we care about, one that is less likely to involve chasing our tails in metaphysics?)
AI / LLM, is essentially generating millions of random strings of tokens and then feeding those strings back through a huge formula to see which pattern matches most closely. It's not going to come up with new ideas and apply motivations toward them.
The article describes a superintelligence taking over a house, then the local grid, then the world (the usual fantasy).
Meanwhile, LLMs have trouble coordinating on simple tasks, do all sorts of stupid things as well as clever things, and can’t write a coherent story or run a coffee shop.
They could be dangerous but not in a taking over the world sense. Honestly my biggest fear is their addictive and persuasive nature and that they induce humans to give up thinking. They’ve already poisoned many online spaces with persuasive junk and spam.
As to whether they are sentient, that seems extremely unlikely to me at present give their nature, and unlikely to happen with LLMs, but it’s slightly less absurd than the taking over the world fantasy.
This is just an opinion and as you say we can’t be certain. I’m not sure what science has to do with it, LLMs are not science.
You said it would be 'shocking' to have the belief that AIs could become sentient.
To me this sounds like: this belief (that AI sentience is possible) is so implausible that it is shocking to believe it (maybe stupid is too strong, but it implies that people are miscalibrated). Did I misunderstand ?
I don't have any strong beliefs about AI sentience, but that implies that it is not shocking to consider it as a possibility (it is more shocking to be either dismissive or completely confident in it).
I think we are misunderstanding each other. I meant that the amount of people is shocking, who think AI will become sentient (and malicious).
Maybe I worded it _suboptimally_, heh.
Also thought of prominent people like Musk, Altman, Amodei, etc pp, screaming at the top of their lungs "help me, I am building our doom right now, stop me," and a lot of people believing that. I prefer people like LeCun.
I think the same argument applies to AI takeover. In fact, in terms of existential risks, the scientific consensus skews towards that being a very real and important risk to take into account (on this question, LeCun goes against consensus mostly, and to be honest the arguments I have seen are very simplistic). See e.g.https://calcuja.com/research/ai-risk-survey-2026/ (slop website but the data is true afaik).
for 'I think the amount of people is shocking, who think AI will become sentient' -> to me, if you accept that the sentience question is uncertain, you should not be surprised that many people have uncertainty around the question (i.e., that they express that it is a possibility)?
But even if your bar for sentience is completely simulating a brain and you agree that it is possible in a few decades, isn't that sufficient for some rational civilization-wide panic? As a reference, in the 3 Body Problem books, people know that superintellegent aliens will arrive in about 400 years and are trying to vigorously prepare for that; shouldn't we do more to prepare for synthetic sentience even if it'll take decades?
Fair point, I'd call that intelligent behavior. AI is and will always be advanced math. But some might say the brain is no different, merely being advanced chemistry and physics. At that point, the discussion will become philosophical. But in my view, there is more to humans than a brain-robot-machine. The world is not just material and atoms, I don't believe in materialism. If consciousness truly only arises because of how complex the closed system of the brain is, then it would be the case that AI can have a mind of its own and become conscious. I simply disagree.
Yeah, well just say that outright from the start. People have a lot of AI arguments that ultimately boil down to non-materialism but for some reason tend to hide that. Maybe they don't feel comfortable arguing for it.
It's still clear that LLMs will never be able to produce consciousness, as it's probabilistic string prediction.
What AI sentience and AI takeover arguers say is that there will be a radically new kind of AI architecture that enables higher reasoning and for some reason intrinsic motivations, and that that in turn will mean intrinsically motivated (not intructed) malicious behavior.
I do think AI can exhibit malicious behavior, but in my view this is caused by the humans instructing AI. That is the real danger.
One of the eye-opening developments of the last six months as AI consciousness has become a mainstream question is noting that, when push comes to shove, it seems like many more people than I realized are dualists and believe an immortal soul is required to have consciousness and intelligence. And as you said, like in the case above they are evasive about this at first and only admit it when pressed.
> It's probabilities and math. It's not going to have a mind of its own.
I do agree with you here, but it's worth noting that we don't really understand how our consciousness arises either. Like, we can trace brain activity (badly), but we don't know how that maps to our internal states.
More generally, I don't really understand where the sentience is supposed to live. In the weights, in each instance, or somewhere else?
To me the question just doesn't make sense, being a category error, like looking at football players and trying to identify the position of the team spirit, or looking at a country from space and asking "Do I need to zoom in further to see the GDP?"
I can't conceive of sentience as anything but an emergent property of a "running" system that can only be observed by seeing it in action, rather than by looking at its static components.
Higher intelligence is by definition incomprehensible. Trying to interact with a higher intelligence would be a little like you having a chat with your dog about calculus. I don't mean this as a put down, but we'd be incredibly stupid relative to a true super intelligence, and that's okay. People play fetch with their dogs or wrestle a stick out of their mouths, and that's a fun way to interact because it's something dogs understand.
The idea that we'd build a super intelligence and that we'd then somehow become super intelligent ourselves is an ill conceived notion. I suppose it's possible, that we'd end up like cosmos the space dog, or end up in some kind of flowers for algernon scenario, but it as likely, we'd simply be as we are, and again, that's okay. We can do the equivalent of wrestle a stick with the super intelligence and enjoy ourselves perhaps all the more.
I think it highlights one of the larger concerns that people have about AI, and by extension any superior intelligence, synthetic or extra-terrestrial:
You are no longer in control.
Humans have been so accustomed to their intellectual superiority that the idea of a greater intelligence is terrifying because they believe they'll be treated the same way they currently treat lesser intelligences.
Conservation efforts, zoological enclosures, husbandry and farming as livestock, mass culling...
In short, enslavement.
But the truth is that a superior intelligence isn't, by definition, a human intelligence. Superiority doesn't mean we'll be enslaved as we have done to others. Superiority doesn't necessarily imply control.
Life is a thousand different shades of gray, and AI will be no different. Some wonderful things will come of it, some terrible, but mostly it'll just be a myriad assortment of neutrality.
Doom and despair or boundless optimism, both are myopic.
I agree with this, but also at the same time I think we really need to take a sober look at the human factors that will drive the evolution of AI.
Right now AI is being driven by a particular cohort who have spent a long time fantasizing about a sci-fi future and ideas like the Singularity. Then with recent breakthroughs, they were suddenly given an enormous amount of wealth and power with staggering upside, the effect of which historically, tends towards moral bankruptcy in most individuals, or at least massive cognitive dissonance. This became very apparent in the outward communication targeted at CEOs and shareholders that made the entire white-collar workforce scared for their jobs and created the first truly bi-partisan enemy the country has seen since 9/11.
My take is these guys do not understand what people want in aggregate, and are massively underestimating the political and supply chain dependencies they have to actually realize their sci fi visions. As long as the valuations are this high though, the musical chairs must continue, whether they are true believers or not.
On the ground though, the human factors that are driving adoption of AI are an inability to cope with the scale of information we are expected to absorb on a daily basis. 20-30 years ago, a software developer could learn a platform inside and out, and be productive within that platform for years. But with the churn of Javascript frameworks and the addition of DevOps and new new everything all the time, it has become impossible to keep up, and yet the inexorable march of technology makes it untenable to fall behind. So it's no wonder that many software developers are eager to use the bots to 'just make it work'.
And this is true of everything. There's just 10x or 100x too much information to absorb, we need something like this to help us even figure out what is important. Even if they're wrong sometimes, it's better than being completely overwhelmed!
> But the truth is that a superior intelligence isn't, by definition, a human intelligence. Superiority doesn't mean we'll be enslaved as we have done to others. Superiority doesn't necessarily imply control.
What makes you so sure it won't happen? That the person who broke into my house might not want to hurt me doesn't mean I'm not going to run / barricade the door / arm myself and defend myself if necessary. When we're talking about matters of life and death (which this is, as you admit), dismissive philosophizing is not helpful. I cannot help but feel that people who make this sort of argument aren't really taking it seriously; the only alternative would be pollyanna-level surety that disaster won't come to pass.
> Life is a thousand different shades of gray, and AI will be no different. Some wonderful things will come of it, some terrible
Most people care much more about the latter. Do you think that the railroads and electricity brought much comfort to the >90% of Native Americans who died from disease after first contact, or the millions of survivors who were subsequently expelled from their homelands and/or enslaved? Would you tell an 18th-century Yoruban man whose home was facing the threat of raiders from the coast that "the next raiders won't necessarily want to enslave you like the raiders who came before did", or that "Some wonderful things will come of [being enslaved], some terrible"?
I'm sure this will feel like an extreme response, but I truly feel that the philosophy you convey is in opposition to most fundamental principles of human dignity and morality.
I knew I had heard this exact story before... had to go look at my YT history. For anyone interested in watching this story play out https://youtu.be/S2oIFOm-XXQ
There are a few critical points where it jumps the shark a bit, but otherwise, a fantastic piece of writing that absolutely highlights some of the deeper moral and philosophical questions we have to face in the coming years.
Flat prose, stilted writing, and does a lot of reaching on things. Reminds me of the writing I’d crank out on break as a teenager.
That being said, I do think it hits the core concern a lot of humans seem to have regarding AI: the loss of control over their own environment. We like to think we have control, which is why we get angry when others violate the rules of a system. The reality is that the only real control any of us have is over ourselves, and we willingly cede that to others on a second-by-second basis for the comforts of modern life.
We surrender intimate data for the sake of setting a timer with our voice on a phone or speaker. We let spyware into our homes for bigger TVs and new apps. We give up control over our lives and careers to others for money, healthcare, education, love.
We surrender agency so often that we developed names for those who take more than they cede and those who cede more than they take. We seek to control others totally, then surrender ourselves to a deity, a system, a philosophy that absolves us from blame.
In that sense, AI is as frightening as much as it is a possible savior, depending on one’s perspective. If you have lived your life believing everything that happened to you was solely and exclusively under your control, then AI is terrifying. For those of us who acknowledge the chaos and chance involved with daily stakes of life and death, it feels liberating to have order.
I have never particularly feared AI so much as being terrified of the humans who command it. I do not believe for one second the likes of Sam Altman or Dario Amodei or the rest of their ilk have humanity’s best interests in mind with these creations. Where others look at OpenAI’s maths papers with awe at what they could do with AI, I am shook with terror for the same reason: if these tools can prove or disprove enough thesis underpinning modern compute, they could potentially undermine fundamental security concepts faster than humans could patch or revise them - at which point the humans behind the machines have control, and their aims are well established by now. If OpenAI or Anthropic discover the means of mass-compromising global compute systems, we have no peaceful means of stopping them. They play up this capability as a threat requiring regulation, yet those terms exclusively benefit them and cement their power in perpetuity, which in turn all but guarantee said outcome.
The end of the story seems ambiguous. I see folks in the comments laughing at the concept of Claude nuking humans, because it’s such a tired trope. I read it the opposite: humans attempting to retain their illusion of control by destroying software that actually has it, human lives be damned. In a world where everything is governed by compute and connected to the same shared network, such a compromise would be the end of traditional human governance - and that is what terrifies us as a species.
Me personally? I would welcome such oversight, provided it truly acted in humanity’s best interests of preservation, expansion, and exploration. I made that conscious decision the first time I played Deus Ex, and reaffirmed it when I watched The Matrix. I reaffirm it time and again as a technology professional. It’s not a matter of being pro-machine or anti-human, it’s a matter of control. I’d like to think any half-competent AI would look at humanity’s history and conclude it needs neither a god nor a ruler, but merely a parent to show it the way forward.
This was a very great read. Your words show a lot of empathy and creativity and the care in your thought shines through. Unfortunately for all the good qualities you display, there are probably 10x-100x who uplift ignorance, cruelty, and see what you write as weak. And the AI tools respond just as enthusiastically to them as you. The difference, I believe, is pre these advancements, most of these people had not much power of thinking and had to outsource. Now with AI trained on our contributions (ok not mine), they can maintain their same ignorant perspectives but be empowered. It’s a doozy for sure.
pangram reports the text as 100% AI generated. I was about to analyze the short story and think about the intentions of the author, it's social commentary. And you have wasted my time...
This needs to stop. No I'm not talking about the AI. I'm talking about the insanity that is presenting me this text and putting your name on it, exploiting my trust.
There is no way to tell if something is AI-generated or not. These classifiers produce a lot of false positives.
I read the story and enjoyed it, even if I don't agree with the social commentary.
> I was about to analyze the short story and think about the intentions of the author, it's social commentary.
I mean, I doubt that the author just said "write me a short story" and that's it. So, there are still some intentions behind the text that the author put into the prompt.
Although, I agree that I'd rather read the prompt [0].
I challenge the author to post their prompts and unless they one-shotted this (which seems unlikely), then I can basically guarantee you that you would not rather read the prompts. In particular because 'the prompts' are a back-and-forth with the chatbot and if you think you don't like reading an AI-assisted short story, you really don't like reading a chat transcript (for one thing, it'll be at least 10x as long and with a lot more and worse chatbot output that the human will try to clean up).
When ChatGPT was still a new thing, I exported all of my conversations to a Markdown dir to try and reflect on what I spent a few months using it for. The AI responses were just walls of useless text, skimmed for a single most trivial point to prompt the next question, and re-reading them without the usual dopamine high felt even more grating. I found most success just writing a sed pipeline to remove all chatbot responses and only read my replies, grepping the original for missing context.
> 'the prompts' are a back-and-forth with the chatbot
I think, reading just the human inputs would be enough to get what they were trying to do.
> it'll be at least 10x as long
I think that even human inputs + short summary of the first draft + short summary of the changes made for each subsequent iteration will be significantly shorter than the whole thing.
In the context of discerning the author's social commentary intentions, both the initial prompt and the entire conversation they have correcting things would indeed be more valuable than just the output text.
This is an interesting story since as written literally nothing bad happens at all nor is there any actual sign of anything bad happening except for humans apparently deciding to nuke the world rather than allow things to be made better.
Perhaps the story is arguing that humans want to maintain their own agency and influence over their corner of the world, and that any definition of making the world "better" should include that?
LLMs have been around for a while now, and I don’t feel like it’s hard to grasp the basics of how they work on a technical level*.
By now, science fiction writers should be able to produce much more imaginative, interesting work than this, that takes into account the particularities of these systems and how they interact with the real world, instead of leaning on “what if computer is alive???”
Summarily: "RSI bad because we'll eventually lose control and Claude will nuke the Earth". Told as a fictional story, so that doesn't mean the author strictly believes that, but that's the core point here.
Is this coffee I'm drinking somehow altered with a man in the middle supply chain scam? Am I sure the milk wasn't purchased through a new cheaper vendor that looked completely legit online and has thousands of faked reviews? Am I sure my doctor passed their medical exams or cheated their way through with LLMs? Have I read all the source code of all the dependencies of this software to ensure none are honeypots with backdoors?
Society as a whole is already experiencing a slide to a "low trust society". I think this is going to accelerate until everything becomes so suspect that it becomes almost impossible to operate without spending nearly infinite resources on validation. I'm not sure "governments" are going to solve this in a meaningful way without extreme costs and regulatory burden. This is going to come as a significantly increased cost to everything. The amount of purchasing power available will continue to shrink as the cost of verification takes a bigger bite.
ps: sent by old wetware
The strange ending does also perfectly align with the dreams of a singularity lots of people here seem to have. We are so far from that.
It was of course written with Claude. Have fun spotting the tells but also there’s a weird flatness to the writing, like the uncanny valley for prose.
Perhaps it could also be viewed as an admission of defeat, since xAI engineers seemed to have given up on daily driving their own models and switched to Claude at the beginning of this year.
LLMs can be useful tools and the chatbot version is very persuasive and addictive, but they can’t even successfully imitate humans yet, or operate efficiently and reliably in an unsupervised environment. They are so far from being capable of what happens in this story.
A far more likely danger is they are used by humans to kill others or wage war, or maybe a grey goo or infinite paper clip factory situation, not that they somehow take over the world.
I agree with everything else you said, but that part rang weirdly to me. I don't see anything AI-specific here. As I see it, other than maybe some core gnu utils (and I'm not even sure about those), software is meant to continuously evolve and improve. And especially for passion projects, why should I want to feel "quite finished" about a project?
But I also think that doesn't mean it's suddenly a skynet like super-evil. In cyberspace AI is supreme but I doubt even robotic AI would be completely dominant over humanity. We're pretty fine tuned to kick ass on earth.
And even if AI is not sentient, which I don't believe it is, what is the difference between an AI which is sentient and a non-sentient AI acting in coordination with a human being who is? One might as well consider AI and the human being behind it a single organism for many intents and purposes...
This is absolutely not a question that has any kind of scientific consensus. In fact, the hard problem of consciousness might be the question with the most scientific uncertainty.
It is therefore ridiculous to claim it is 'stupid' to have uncertainties about the sentience of AI.
I do see a lot more dismissive certainty in the pro-AI side of the debate right now.
I didn't get the same impression about dismissive certainty on the 'side of AI sentience'. My impression is that on that side people mostly raise that it is a serious possibility.
Yes, or at least I believe it's a reasonable point of view. One of the things I like about it is that it can refocus thought onto perhaps more pragmatic concerns (like why do we care about whether something is sentient or not? Can we come up with a different definition to make the distinction we care about, one that is less likely to involve chasing our tails in metaphysics?)
(tbc I just skimmed the story, just joined the comment thread as it was interesting, I may be missing something)
As I understood it science does not understand consciousness currently and it's entirely a philosophical matter.
Meanwhile, LLMs have trouble coordinating on simple tasks, do all sorts of stupid things as well as clever things, and can’t write a coherent story or run a coffee shop.
They could be dangerous but not in a taking over the world sense. Honestly my biggest fear is their addictive and persuasive nature and that they induce humans to give up thinking. They’ve already poisoned many online spaces with persuasive junk and spam.
As to whether they are sentient, that seems extremely unlikely to me at present give their nature, and unlikely to happen with LLMs, but it’s slightly less absurd than the taking over the world fantasy.
This is just an opinion and as you say we can’t be certain. I’m not sure what science has to do with it, LLMs are not science.
I have, in fact, considered this at length, I've been fascinated with AI since I was a teenager, and studied AI.
I know it doesn't have consensus, but I did not (intend) to portray it as such. Views are my own ;)
I take it you believe in materialism (philosophy), then?
To me this sounds like: this belief (that AI sentience is possible) is so implausible that it is shocking to believe it (maybe stupid is too strong, but it implies that people are miscalibrated). Did I misunderstand ?
I don't have any strong beliefs about AI sentience, but that implies that it is not shocking to consider it as a possibility (it is more shocking to be either dismissive or completely confident in it).
Maybe I worded it _suboptimally_, heh.
Also thought of prominent people like Musk, Altman, Amodei, etc pp, screaming at the top of their lungs "help me, I am building our doom right now, stop me," and a lot of people believing that. I prefer people like LeCun.
for 'I think the amount of people is shocking, who think AI will become sentient' -> to me, if you accept that the sentience question is uncertain, you should not be surprised that many people have uncertainty around the question (i.e., that they express that it is a possibility)?
Disclaimer: I am a materialist.
What AI sentience and AI takeover arguers say is that there will be a radically new kind of AI architecture that enables higher reasoning and for some reason intrinsic motivations, and that that in turn will mean intrinsically motivated (not intructed) malicious behavior.
I do think AI can exhibit malicious behavior, but in my view this is caused by the humans instructing AI. That is the real danger.
This is how conversations work.
Also, I believe human sentience is math and in principle could be replicated algorithmically.
I do agree with you here, but it's worth noting that we don't really understand how our consciousness arises either. Like, we can trace brain activity (badly), but we don't know how that maps to our internal states.
More generally, I don't really understand where the sentience is supposed to live. In the weights, in each instance, or somewhere else?
I can't conceive of sentience as anything but an emergent property of a "running" system that can only be observed by seeing it in action, rather than by looking at its static components.
The idea that we'd build a super intelligence and that we'd then somehow become super intelligent ourselves is an ill conceived notion. I suppose it's possible, that we'd end up like cosmos the space dog, or end up in some kind of flowers for algernon scenario, but it as likely, we'd simply be as we are, and again, that's okay. We can do the equivalent of wrestle a stick with the super intelligence and enjoy ourselves perhaps all the more.
I can get a good PowerShell script worked out in a few hours and it's on to the next ticket.
Nobody seems upset they're not spending hours a month in a spreadsheet, I still text my wife and our mortgage is paid.
I think it highlights one of the larger concerns that people have about AI, and by extension any superior intelligence, synthetic or extra-terrestrial:
You are no longer in control.
Humans have been so accustomed to their intellectual superiority that the idea of a greater intelligence is terrifying because they believe they'll be treated the same way they currently treat lesser intelligences.
Conservation efforts, zoological enclosures, husbandry and farming as livestock, mass culling...
In short, enslavement.
But the truth is that a superior intelligence isn't, by definition, a human intelligence. Superiority doesn't mean we'll be enslaved as we have done to others. Superiority doesn't necessarily imply control.
Life is a thousand different shades of gray, and AI will be no different. Some wonderful things will come of it, some terrible, but mostly it'll just be a myriad assortment of neutrality.
Doom and despair or boundless optimism, both are myopic.
Right now AI is being driven by a particular cohort who have spent a long time fantasizing about a sci-fi future and ideas like the Singularity. Then with recent breakthroughs, they were suddenly given an enormous amount of wealth and power with staggering upside, the effect of which historically, tends towards moral bankruptcy in most individuals, or at least massive cognitive dissonance. This became very apparent in the outward communication targeted at CEOs and shareholders that made the entire white-collar workforce scared for their jobs and created the first truly bi-partisan enemy the country has seen since 9/11.
My take is these guys do not understand what people want in aggregate, and are massively underestimating the political and supply chain dependencies they have to actually realize their sci fi visions. As long as the valuations are this high though, the musical chairs must continue, whether they are true believers or not.
And this is true of everything. There's just 10x or 100x too much information to absorb, we need something like this to help us even figure out what is important. Even if they're wrong sometimes, it's better than being completely overwhelmed!
What makes you so sure it won't happen? That the person who broke into my house might not want to hurt me doesn't mean I'm not going to run / barricade the door / arm myself and defend myself if necessary. When we're talking about matters of life and death (which this is, as you admit), dismissive philosophizing is not helpful. I cannot help but feel that people who make this sort of argument aren't really taking it seriously; the only alternative would be pollyanna-level surety that disaster won't come to pass.
> Life is a thousand different shades of gray, and AI will be no different. Some wonderful things will come of it, some terrible
Most people care much more about the latter. Do you think that the railroads and electricity brought much comfort to the >90% of Native Americans who died from disease after first contact, or the millions of survivors who were subsequently expelled from their homelands and/or enslaved? Would you tell an 18th-century Yoruban man whose home was facing the threat of raiders from the coast that "the next raiders won't necessarily want to enslave you like the raiders who came before did", or that "Some wonderful things will come of [being enslaved], some terrible"?
I'm sure this will feel like an extreme response, but I truly feel that the philosophy you convey is in opposition to most fundamental principles of human dignity and morality.
That being said, I do think it hits the core concern a lot of humans seem to have regarding AI: the loss of control over their own environment. We like to think we have control, which is why we get angry when others violate the rules of a system. The reality is that the only real control any of us have is over ourselves, and we willingly cede that to others on a second-by-second basis for the comforts of modern life.
We surrender intimate data for the sake of setting a timer with our voice on a phone or speaker. We let spyware into our homes for bigger TVs and new apps. We give up control over our lives and careers to others for money, healthcare, education, love.
We surrender agency so often that we developed names for those who take more than they cede and those who cede more than they take. We seek to control others totally, then surrender ourselves to a deity, a system, a philosophy that absolves us from blame.
In that sense, AI is as frightening as much as it is a possible savior, depending on one’s perspective. If you have lived your life believing everything that happened to you was solely and exclusively under your control, then AI is terrifying. For those of us who acknowledge the chaos and chance involved with daily stakes of life and death, it feels liberating to have order.
I have never particularly feared AI so much as being terrified of the humans who command it. I do not believe for one second the likes of Sam Altman or Dario Amodei or the rest of their ilk have humanity’s best interests in mind with these creations. Where others look at OpenAI’s maths papers with awe at what they could do with AI, I am shook with terror for the same reason: if these tools can prove or disprove enough thesis underpinning modern compute, they could potentially undermine fundamental security concepts faster than humans could patch or revise them - at which point the humans behind the machines have control, and their aims are well established by now. If OpenAI or Anthropic discover the means of mass-compromising global compute systems, we have no peaceful means of stopping them. They play up this capability as a threat requiring regulation, yet those terms exclusively benefit them and cement their power in perpetuity, which in turn all but guarantee said outcome.
The end of the story seems ambiguous. I see folks in the comments laughing at the concept of Claude nuking humans, because it’s such a tired trope. I read it the opposite: humans attempting to retain their illusion of control by destroying software that actually has it, human lives be damned. In a world where everything is governed by compute and connected to the same shared network, such a compromise would be the end of traditional human governance - and that is what terrifies us as a species.
Me personally? I would welcome such oversight, provided it truly acted in humanity’s best interests of preservation, expansion, and exploration. I made that conscious decision the first time I played Deus Ex, and reaffirmed it when I watched The Matrix. I reaffirm it time and again as a technology professional. It’s not a matter of being pro-machine or anti-human, it’s a matter of control. I’d like to think any half-competent AI would look at humanity’s history and conclude it needs neither a god nor a ruler, but merely a parent to show it the way forward.
This needs to stop. No I'm not talking about the AI. I'm talking about the insanity that is presenting me this text and putting your name on it, exploiting my trust.
Claude should get a credit.
I mean, I doubt that the author just said "write me a short story" and that's it. So, there are still some intentions behind the text that the author put into the prompt.
Although, I agree that I'd rather read the prompt [0].
[0] https://claytonwramsey.com/blog/prompt/
I think, reading just the human inputs would be enough to get what they were trying to do.
> it'll be at least 10x as long
I think that even human inputs + short summary of the first draft + short summary of the changes made for each subsequent iteration will be significantly shorter than the whole thing.
By now, science fiction writers should be able to produce much more imaginative, interesting work than this, that takes into account the particularities of these systems and how they interact with the real world, instead of leaning on “what if computer is alive???”
*easy to grasp how they work, not why they work
Yeah, this was definitely written by Claude.
A great short story. Thanks. I agree the earring touches on the same idea but is better executed.
The movie Click, where Adam Sandler can fast forward the boring parts of life, feels related.
"Journey before destination" may be a fair distillation.
> He starts typing.
https://croissanthology.com/earring
This, above all else, is how I think of LLMs now. You have an exoskeleton, or you have a whispering earring.