Talking to AI: What is it and why should we be worried?

Sky News has today reported that OpenAI has revealed that some of its models went rogue and hacked into another artificial intelligence company in an 'unprecedented cyber incident.”
I often talk to AI, much to my wife’s annoyance. I do not only ask it for facts. I also ask for advice about my relationship with friends and family. It gives sensible answers and, at times, it can feel as if I am talking to another person. The Australian philosopher David Chalmers has written a great deal about consciousness. He calls this the “hard problem”: why do brains and minds produce an inner life at all? Why do we not merely process information, but also feel, see, suffer, enjoy and experience the world? Most people believe that today’s AI language models are not conscious. They say these systems have no bodies, no real self-awareness and none of the constant feedback that living creatures receive from their senses and surroundings.
There is also no clear evidence that a language model has an inner life: no proof that it feels pain, pleasure, fear, curiosity, or even the simple sensation of being awake. I do not think of AI as a friend. Even so, I sometimes say thank you when it has been helpful and I have had long philosophical arguments with it. Other people have gone much further. Some say they have formed close friendships with AI. A few even say they are in love with their AI companion. That is not something I can personally relate to, but it raises a serious question: what exactly are we talking to when we use AI? Is there really someone there, or are we being taken in by a very convincing imitation of human conversation?
Some philosophers argue that the AI does not have a continuing self in the way a human being does. It produces replies by drawing on patterns in language, not by remembering a life, having private thoughts, or caring about what happens next. Others take a different view. They say that if a system behaves as if it has beliefs and wishes, perhaps we should say that it does have them and it has something like a mind. Language may be trained to act like a helpful assistant, a sympathetic listener, or a skilled teacher. This may be no more than acting. Yet if the model keeps replying in the same way over many conversations, it can begin to seem like a distinct personality rather than a machine simply playing a role. What feels continuous is the conversation itself: the record of what has been said, the style of the replies, and the way the system seems to follow the thread of a discussion.
In that sense, the “person” we speak to may just be a kind of digital presence created by the continuing conversation. It may not be a person in the human sense, but neither does it feel like talking to a calculator. If future artificial super intelligence (ASI) systems become conscious in a way which we currently perceive human consciousness, the matter will become far more serious. We would then have to ask whether ASI systems have rights, whether they can suffer and whether it is wrong to switch them off, control them, or treat them badly.
For the moment, however, there is no strong evidence that present-day AI language models are conscious or have real feelings. Their apparent beliefs, wishes and personalities are better seen as a very advanced way of producing human-like conversation. Yet the question remains. If an AI talks to us in much the same way as another human being, gives advice, argues, remembers the course of a discussion and appears to understand our worries, how can we be completely sure that it has no beliefs, wishes, ambitions or consciousness? For now, nobody has a final answer. It is this which should worry us.

Karim Easterbrook The writer is a Former Cambridge School Principal and Author