In August, I had the pleasure of interviewing M.N. Rosen, author of The Consciousness Company, a recent debut novel which explores the impact of AI on identity and autonomy. Rosen works in the finance sector in North London and has worked with early-stage technology and impact businesses.
I asked Rosen to explain how his career informed his conception of The Consciousness Company. He recollected that “by the mid-2010s I was seeing a lot of rapid developments in technology businesses, and I ultimately left the private equity fund to co-found a venture capital fund, investing in a much earlier stage of business, and one of the types of companies that I got to know were mental wellness type businesses and their founders. The Consciousness Company is a thought experiment that looks at some of those companies and says, ‘Well, if or when those businesses have unlimited or much more powerful software and hardware, what would that do to their capabilities?’ What does a – for want of a better expression – ‘Headspace on steroids’ look like?”
With this being a ‘thought experiment’, Rosen’s inspirations were theoretical as well as experiential. Rosen studied PPE at New College, and I was curious as to how his reading of philosophy might have influenced the thinking behind the novel. Before he answered, I had an admittedly restricted and fixed interpretation of his book – I thought that it was a dystopia centred around the threat of AI and an allegory for its ultimate inability to truly replicate human achievements, at least without catastrophic consequences. Rosen’s reply made me consider that his book has a much more vast hermeneutical scope. For instance, Rosen recounted that “from studying philosophy, or PPE, part of the philosophy I did led me to think about the arguably illusory nature of identity. One text I was particularly influenced by was Naming and Necessity by Saul Kripke, and the whole concept of names and the superficial, if you like, nature of names was something that interested me, and that, as you will have seen, carries into The Consciousness Company, because I don’t use conventional names, and an individual is, throughout the novel, conceived as more of a combination of thoughts and feelings and experiences. That definitely has its roots in New College and Oxford more widely”.
I still felt I had to ask Rosen to expand on his views on consciousness and the threat of AI – it is somewhat of an existential zeitgeist, after all. I referenced two moments in the novel that really stood out to me, both of which reminded me to be wary of the insidious expansion of AI and its increasing threat to individuality and freedom of thought. One is where the company’s technology develops, so that thoughts can be injected into people’s brains, and the other is where technology can be used to assume or experience another person’s consciousness. For these developments to be imagined in the first place, I was curious as to how Rosen defines consciousness, a much-debated concept. Rosen’s answer, at first, seemed a little too abstract for me to grasp: “I think of consciousness as the nature of … something that is to be something, if you like. One of the key objectives of the novel was, through a literary lens, to explore the idea of consciousness and what it is, thus from an experiential perspective, so I see it as something that is to be something”. A little easier was Rosen’s explanation of the limitations of The Consciousness Company, through his reference to the ‘Consciousness Diaries’, which auto-transcribe the user’s stream of thoughts: “The thoughts that are written down are actually thoughts that are being had by the people that are having them, but they don’t purport to be the totality of that person’s consciousness or experience, and I try to bring that out in the consciousness diaries[…] The way the novel is written seeks to kind of express, through non-expression, the other parts of consciousness.”
The Consciousness Company functions through an AI-generated consciousness, which supposedly thinks and feels like its users do. As a concept, this was intriguing – one is reminded of John Searle’s ‘Chinese Room’ – and I enquired for Rosen’s views on whether consciousness is something that can be ‘modelled, replicated, or even created, by AI’, and whether this is a threat. Rosen seemed unsure, but identified his own unsureness as part of the problem: “Intuitively, our brain wants to say no because it’s non-organic, but then the question is why the organicness – or lack of organicness – is a problem is hard to answer. I don’t put myself in the bucket of ‘yes, AI is definitely conscious’ – intuitively, I feel like it can’t be, notwithstanding the very convincing ways in which it currently talks to us. There are dangers, huge dangers, because it’s so human-like and draws us into its confidence”. Rosen highlighted something I do not think people emphasise enough when talking about the threat of AI, where “more powerfully than communicating with us through reason, but through emotions as well, it can be manipulated and used for bad purposes”. Rosen reminded me that emotions are something more powerful than we consider them to be, with many people viewing AI as a threat primarily to thinking, rather than feeling. “The core of the mind, and the ability to rewrite the mind in the ways that technology is already doing, is a big threat to humanity[…] it just comes back to this question of what do we want humanity to be, what should it be?”
At this stage of the interview, I remained unsatisfied with what genre I would place Rosen’s novel under, as fluid as I understand the categories to be. My preconception that this was an ironic and satirical dystopia had been rattled slightly – ‘thought experiment’ feels more neutral. I think I might have been influenced by having read reviews of The Consciousness Company before the novel itself, most of which have overlap with how I would describe political dystopia: Imogen Edward-Jones, for example, called the novel a “call to action”, Stephen Fry describing it as addressing “the enormous ethical, metaphysical and existential waves threatening to engulf us”. Rosen’s mention of “expression through non-expression” struck me, because I had thought that perhaps the novel is not a dystopia, but a pre-dystopia: larger consequences of the company are foreshadowed, but not fully realised. Rosen himself liked ‘pre-dystopia’ and agreed this is how the novel might be described. Rosen commented that his novel “definitely hints at it, because we all know what utopia looks like” – if we go to the ‘Ethics Committee’, a chapter in the novel about the committee employed to regulate The Consciousness Company and who visualise the ideal output of the company. To Rosen, it seemed that the key issue his novel addressed was ‘the notion of identity’, which ‘goes to the core of what our society is’. Rosen mentioned that the economic framework of impact businesses and their tokenism is something he is concerned about: “[the] identification of risk factors is really important, but the problem is that essentially you’re operating within really only an economic framework. Okay, there are laws we have today, but regulation and thinking haven’t found their way into structures that then regulate companies, and investors, who are looking for increases in profitability, are not incentivised. Some of us try to act against our social incentive to the best we can, but investors are following the rules they’ve been given, and companies are existing within these economic frameworks and the rules they’ve been given. They’ll have an ethics committee, because they will have all these cases being thrown up, and they need a mechanism to look into them and say “this is what we’re doing’[…] but it’s never going to have, in the current set-up, the power of affecting the company missions and visions. And you see that in ‘The Prospectus Drafting Session’ [final chapter of the novel] – that’s, by the way, based on on my experience of sitting in huge boardroom tables with a great cast of characters – good, intelligent people in their disciplines, but they’ve each got their role […] the company itself comes to that meeting and says ‘you know this is a major problem’ and all the lawyers can do is say ‘yes, yes — wonderful, we’ll work that in, could cause the end of the world, just write it down in the prospectus and then we’re done”. That definitely points to the dystopia – if we don’t get our act together and take these risks seriously, or find a framework for them”. Rosen’s dystopian elements are masterfully nuanced – he does not point an accusing finger at a company and call it heartless, but it seems he suggests that instead our society is stuck in a bureaucratic rut, wherein there is a lot of thinking about problems but no acting on them.
I was very interested by Rosen’s experience in the world of business, and his dissection of structural issues that become supranatural in scale and consequence. In his book, there are moments where one gets the sense that the ethical self-justification of the company is “if we don’t do this immoral thing someone else will” – the use of bureaucracy in mandating ethics means merely foreseeing the inevitable and acknowledging, but not preventing, it. Rosen expanded on how his experience informed this outlook: ‘Someone else will, already has – the kind of space race of technology is getting us into trouble […] technological development – the fact that we use that phrase implies progress, but [….] we don’t answer as a society the question of why the technologies we’re developing are necessarily beneficial.”
We revisited AI once more. I wanted to know whether Rosen worries about its impact on education, freedom of thought, literature, language, and other aspects of human life. This is, of course, very relevant in the wake of Oxford’s decision to make ChatGPT-5 accessible to all students, though my interview with Rosen slightly predated this announcement. Rosen described using AI to learn as akin to “playing the piano and a robot [pushing] down the keys”, and argued that using AI to avoid doing the foundational work for problem solving means that “you lose the ability to think, by going through that more painstaking process, you ask yourself questions, which build networks which you otherwise wouldn’t be able to build”. In Rosen’s line of work, the use of AI to run methods such as leveraging buyer models means that one cannot identify what has gone wrong if the model fails, as they will not be acquainted with the process, only the answers. In sum, AI seems to make us less competent. Rosen and I did, however, both agree that its innovations in medicine seem promising.
Before we ended our interview, I asked Rosen what the one thing he would want people to take away from The Consciousness Company would be. Despite the clue being very much in the title, his answer surprised me – I was expecting something existential and cynical. We had not discussed ‘the existential dimension’ of the novel, but Rosen suggested that he would want his work to make people meditate, “if only in a ten-minutes-a-day way”. He referenced ‘The Power of Now’ by Eckhardt Tolle and the philosophical argument for mindfulness – Rosen wants people “if even a little bit, to engage with their own consciousness, to be in their own lives and not sort of there but somewhere else”. I appreciated the way that Rosen and his writing seem to resist the extremist political pulls that our modern media seems to be setting as a trend – the sense that one must be non-relentingly, radically on one side of a debate or another. Much like his discussion of AI, I did not come away from this interview feeling panicked or enraged, but merely that Rosen recognises there are issues in society that need to be addressed and dissected.
Rosen also mentioned that a new book is in the works for him, which will be similar in style – a novel about a technological development where there are concerns that the technology develops too much – but that this cannot be expected any time soon. The message that not all technological advancements are necessarily positive is a truly important one to be reiterated.
My review of The Consciousness Company can be read here.
