How our time online is changing (ft. CEO of Microsoft AI, Mustafa Suleyman)
Hello everyone, I’m here with Mustafa Suleyman. You’re young, but when you were even younger, you became deeply intertwined with the start of AI’s modern era. You were one of the three co-founders of DeepMind, acquired by Google in 2014. In 2022, you co-founded Inflection AI, which recently in March of 2024 hit a unique deal with Microsoft, where they are able to use Inflection’s models. They hired most of the startup staff, including you, and you’re now appointed CEO of Microsoft AI.
You know this, but this is for my audience. I’m like, I’m telling your life story. It sounds so much more dramatic than it feels. I was going to say, well, my question with that is, was any variation of this path clear to you 15 years ago?
Interesting. Yes, many variations were possible. I definitely think I set an intention very early on in 2009 to be deeply involved in technology. I kind of saw the rise of Facebook in sort of 2007, 2008, 2009, and I could see that it was just mind-blowing how quickly it was growing and really changing the way people communicate, connect, and stay in touch. The subtlety of how the platform shaped what we shared, not just how we shared it, but the type of content was changing.
It was a bit of an aha moment for me that I really kind of realized I want to be part of this massive change. At the time, it just felt like digitization, but then I quite quickly realized that the ultimate version of that is teaching machines to learn and hopefully solve important challenges in the world, making the world a better place. That was my kind of… yeah, I was pretty young when I was 24.
That’s wild. Your current standing at Microsoft, when the average consumer thinks of Microsoft, they probably think of the multi-billion dollar investment that was made into OpenAI back in 2023, which was a continuation of the partnership that started in 2019. With recent provisions, the partnership is set to run through 2030 in its current form.
So for the average consumer, can you clarify how they should view the relationship between Microsoft and your chat experience, co-pilot, alongside OpenAI and their chat experience, ChatGPT? To what degree are they intertwined and to what degree are they competitors?
Yeah, great question. Microsoft is 50 years old, so it has gone through all the major technology transitions of our lifetimes, from the invention of the GUI, the graphical user interface, to the operating system, to going from mainframes to laptops and then back into the cloud. It’s basically been there for all of it, and you know, at its heart, Microsoft is an enterprise company, even though we have a pretty large consumer business with hundreds of millions of users every day of Windows and Word and Excel and so on.
But the part that didn’t quite get as much attention was the consumer AI side, and that’s basically what I came in to run. We have a whole ton of different AI models under the hood at the company, but the biggest one and the most widely used by a long way is OpenAI’s models. I think it was kind of a genius insight by Satya at the time in 2019 when they made the investment. They put a billion dollars into a nonprofit structure before it had delivered any LLM models, GPTs, or meaningful results.
In part, that was a bit of a response to DeepMind doing so well. You know, 2016, 2017, 2018, we had a string of breakthroughs in the game of Go, and then in protein folding, and various other publications and pieces of work. I think there was a general sort of anxiety that this AI wave was starting to crest, and they needed to back a major horse.
I think that takes a lot of courage because typically, you know, a company would think, “Oh, we’ll back our own team internally.” I think they recognized that there was a risk of suffering the innovator’s dilemma if they didn’t spread bet and both back core Microsoft research but also this amazing partnership.
As mentioned, you’ve had your hands throughout your career in a variety of the most impactful AI research labs and companies. Talking about OpenAI again, they uniquely are described as a lab that also operates like a company, which is a unique hybrid model in the field. It seems like with commercial pressures and how that increases, it’s becoming harder for research labs to maybe stick to their mission.
What’s on your radar when it comes to that tension, and how can the value of research labs be preserved?
Yeah, super interesting question. I think it’s important to remember that they were, first and foremost, those labs: DeepMind, OpenAI, and then subsequently Anthropic. There’s not really much precedent for commercial entities. I mean, we fundamentally did found DeepMind as a commercial entity pursuing research almost for its own sake, and so that in itself was kind of unprecedented.
I think over time, as OpenAI realized they have to make money and they have to scale their impact and have a sustainable revenue model, they’ve started to pivot. When a technology emerges at different times during its evolution and development, different types of structures are appropriate.
Where it’s very open-ended, you think about the number of significant moments in OpenAI’s history where there were big pivots. They stopped working on gaming, for example. They stopped working on robotics. They stopped working on simulation. These were sort of one- to two-year major bets for the entire company, and it was really only that last bet of LLMs, which really they took from Google, in the way that we all take from each other in research.
So it’s not a bad thing; it’s just how it works. Google put out this transformer paper in 2017. No one really paid that much attention to it. By 2019 and 2020, they had started turning—OpenAI had turned those ideas in that paper into the first versions of GPT-2 and GPT-3 subsequently. That’s just how research works, but once things really start to work, then the structure that’s needed changes.
You sort of need more engineering, you need over time more sales, and so on, and that’s how you end up scaling. I think everything you just went over really hits a lot of the foundational things right now and the basics for people.
Before we get into the bulk of the conversation, I do want to address a recent situation. I think it would be inappropriate for me not to. The other day at the Microsoft 50th anniversary co-pilot event, both your session as well as the simultaneous interview between Bill Gates, Satya Nadella, and Steve Ballmer were interrupted by pro-Palestine protesters because recent investigations revealed that Microsoft provided cloud computing and AI services directly to Israel’s defense establishment.
In the moment, you responded with, “I hear your protest.” Thank you. While I have to be upfront with my audience that I’m not super well-versed in this relationship, and I don’t feel like I can do the nuance back and forth I typically pride myself on, I do just want to give you space to further comment on the situation so individuals and experts can interpret.
Yeah, I mean, thank you for bringing it up. I acknowledged it in the moment and showed that I respect it and give it space. I think in a world today where tensions on every front are heightened, anxieties are through the roof wherever we look, and often we approach one another with a kind of dismissive and more polarizing tone. That’s just not really within my nature.
I gave it the space that I felt that it needed at the time. It’s a really tough moment. We provide cloud services to many, many different governments and organizations across the world, and they enable all kinds of use cases, just as your smartphone is used for good and for bad.
It’s a very tough moment. We’re all in a very difficult time, and I think it’s a heavy weight that people bear. Being heard and making our case and debating those things openly is a good thing. I appreciate more context.
Another timely topic is the tariffs that were announced last week by President Trump. An immediate reaction from people was being baffled by how these were calculated, even being applied to uninhabitable islands. Upon investigation, X/Twitter users found that ChatGPT, Gemini, Grock, and Claude all recommended some versions of the calculations if someone were to put in a very oversimplified prompt.
Dominic Preston from The Verge said, “If you asked the LLMs for an easy way to solve trade deficits and put the US on an even playing field, they give you a version of this formula: deficit divided by exports with remarkable consistency.” The White House denied any use of AI for the calculations, but in theory, I find two things interesting about this.
One, the importance of good prompting. And two, as someone jokingly commented on a YouTube video I saw, that President ChatGPT arrived sooner than expected, which again is a joke as there’s no autonomy in this situation. But still, in theory, what’s on your mind about this use case?
Yeah, I mean, I doubt very much has been generated with any AI, but you know, we don’t know. I always think it’s dangerous to speculate. But I think it’s more interesting to try and understand the sentiment and motivation behind the tariffs. I understand that there’s fear and anxiety that folks are looking ahead at the next few decades and not seeing a path to sustainable employment.
We’ve already moved a lot of manufacturing outside of the country over many decades, and net-net that’s been amazing for the consumer. We’ve seen better quality products at lower prices. So I think it’s kind of important to try and be empathetic and forgiving for the underlying motivation, whatever you think of the effectiveness of the policy.
Right now, the vote by markets seems to be that they’re not thinking this is going to be a great solution to things, but we’ll see. I more so want to focus on giving the White House the benefit of the doubt, assume that they’re being honest, and that AI wasn’t used for the calculations.
You’ve mentioned in interviews prior that you don’t believe AI should autonomously participate in elections and that democracy is for human participation. But then we think about how the decisions of AI can be communicated through individuals. It’s more so just like in theory, thinking about the importance of good prompting and all that different stuff for these use cases.
It’s really interesting because I mean we’re using AI all the time, and we’re using it to transcribe our speech when we dictate into our phone and generate new images and even search over our images as it looks for cats and dogs in our photos. I think it becomes sort of second nature quite quickly, and suddenly it’s everywhere and integrated.
I do think it’s important to recognize that, largely speaking, it’s been pretty effective. The harms have been surprisingly few and far between. It’s not to say that there aren’t any and it’s not to say that there won’t be any in the future. We shouldn’t be complacent and so on.
But, you know, just consider for a minute, three years ago when these models started coming onto the scene. They were producing all kinds of made-up nonsense all the time, super biased, really toxic, very difficult to control. Prompting wasn’t really even a discipline. There were certainly some people in the field who thought that the bigger they got, the crazier they’ll get, you know, the more unhinged and the more factually inaccurate they’ll get.
Actually, it turned out to be the opposite. The bigger that they’ve got, the easier they are to steer, and now we have many labs purely focused on trying to design the models to follow instructions accurately so that whatever you put in the system, prompt in the prompt itself, or even in the fine-tuning data mix, it’s really good at sticking to those rules and learning from imitation.
That’s actually a very encouraging sign, and I think it’s quite likely to hold so far so good. We see it even more now with these reasoning models, which are the next phase, where because they have looked at the essence of logic and they’ve solved lots of puzzles and they’ve looked at lots of math, they’ve learned the idea of logical structure.
They can take that idea and apply it to domains that don’t have that kind of explicit logical structure. That sounds a bit abstract and a bit nerdy, but I think it’s quite an important intuition to try to grasp, which is it’s learned the essence of logical reasoning. It’s such a powerful fundamental tool, and to the extent that it has that, and it’s good at following instructions, that’s a very useful and capable system.
I think a lot of people assumed there would be a worrying moment with AI in the past election, just like coming into 2024, but there ended up not being a situation, which is again an encouraging sign. There were some generative AI moments, but luckily stuff like community notes helped actually identify that stuff.
That was encouraging in that way. It’s funny how people often say people overhype things short term and underhype things long term. Yes, I hear that. It’s a really good phrase. It seems to apply. People did get a little bit carried away in last year’s elections, but they may have called it too early. They weren’t fundamentally wrong.
I do think over a decade’s time frame, they really do have that problem. During my lifetime, I feel like the power of tech companies has been in my face on a daily basis more than the government’s. Of course, the government decisions are in the background, and things you feel slowly throughout your life, but the power of tech companies is in our face every day.
Back in January, that became very physically real as tech leaders got a front-row seat at the inauguration. What do you see as the most effective relationship between government and tech leaders so that there is a proper level of accountability? Despite tech leaders’ control and deep understanding of these new deeply intelligent tools, what do you think?
I think the good news is that governments have done a pretty good job of adapting. They have educated themselves before this big explosion of LLMs. In 2020, 2021, and 2022, there were lots of efforts to try and understand the implications of the technology. Lots of papers were written, lots of panels convened.
You can dismiss those as bureaucratic blah blah blah, and people often do, but they’re actually really important for helping civil servants, whose full-time job is not to build these models, but to try and understand them. I think that’s really encouraging. I feel like everyone’s having the same conversation now, and that’s also not necessarily true in previous generations.
I feel like during social media, governments were much slower to respond and really understood the consequences much later in the pipeline. They still haven’t really responded. It’s really hard to know how to respond. Sometimes we can see things not quite working, but it takes a while for the sort of regulatory structures to catch up.
It’s not as obvious as it was with seat belts in cars, which themselves took decades after seeing years and years of things go wrong. It’s very obvious what was going wrong. But I think that our cycles of learning as a society are getting shorter and better at learning fast and adapting quickly.
Exactly to that point, your book, The Coming Wave, which I have behind you, focuses on this notion that in the history of all technology and tools, anything that’s useful and valuable gets easier to use, cheaper, and proliferates.
These waves of history are speeding up as each one is amplified and accelerated by the last. Yesterday it was announced that Llama 4 was launched by Meta, and they are again making this an open-source model. Right now, I see in the comment sections the general sentiment that there is a lot of positive notion towards open-source and open-weight models.
It’s applauded because it allows for more transparency into these LLMs and allows for community involvement in audits. Talking about the past decade, 15, 20 years of social media, these blackbox algorithms, there’s been a lack of transparency because we don’t have that access.
People kind of consider these algorithms sketchy. The thought of open-source models is refreshing to people. But as you point out in your book, as these models become more capable, there’s a risk in embracing open source because of that proliferation that can occur. A bad actor, I don’t know, someone in their basement can have access to this really powerful technology, and it can cause some things that are not so good.
Do you think the current trend of companies open sourcing their models will be short-lived? This is newer in the past five years. At what point will companies like Meta, DeepMind, and so on maybe need to revert back to a closed-source approach?
Well, the interesting thing that’s happened in the last sort of six to twelve months is that the absolute frontier, like the very best models in the world, have become available to everybody for free in open weights models or open-source if you like. That is kind of incredible. That’s not really happened at any time in scientific history that the best capabilities are available to everybody for free to build on.
So on the one hand, that’s pretty amazing because creativity can come from anywhere, and now everyone’s using the same tools. Knowledge is spreading faster than ever before because it’s almost like a swarm of people collectively figuring out the problem and then sharing their insights on social media and so on.
That’s an exact comment I saw. Someone was like, this is like a perfect example of utilizing science in the ways it should because, again, the average person has access to it. That’s a really amazing thing, and it’s why I think open source is not going away and is going to keep accelerating at pace. Many of the really significant contributions to the field have come from open source as well.
But obviously, these models are really powerful. We don’t always know what they’re going to do in the real world ahead of time before they’re put into production. Putting a structure around that to try and preemptively catch the things that could potentially go wrong seems like a sensible thing to do.
Right now, companies and labs—the bigger labs—have those obligations. They have shareholders, they have boards of directors, they’re governed by all sorts of other regulations, so it puts a little bit of a constraint on how they operate, which is a good thing. It adds kind of friction.
For example, we proactively report to both the US government and the UK government when we’ve achieved a certain threshold of model performance for a given size. There are lots of small steps that happen which are pretty good. You don’t have that in open source. It’s kind of uncharted territory. We don’t really know how that’s going to turn out.
If somebody uses it for really bad things, they’ll just be able to put it straight out there and everybody else will get access to it. But at the same time, people argue, like you said, having access to it gives people a chance to scrutinize it and test it, break it, try and make it better, and make it more resilient. There are lots of examples, particularly in the security industry, where disclosing a vulnerability in a system and giving everybody the chance to patch that vulnerability in their system, but finding solutions collectively to try and fix the issue—those kinds of things have worked very well and are good for safety and security.
Why do you think social media algorithms are still closed source? I know Elon made the ex-Twitter one open source for a short bit of time, but to this day, why do you still think it’s the case? an interesting question even if the core algorithm was open sourced. I’m not really sure that without the training data, it would be useful. I’m not really sure what you would learn about it. I mean, most people do know how these collaborative filtering algorithms work or these recommendation algorithms work. I mean, there are entire fields of academic research dedicated to optimizing them.
Gotcha, okay. CEO of Microsoft Satya Nadella says he doesn’t believe our furthering relationship with AI will be a winner takes all situation in terms of one single company ruling all, despite there being already a clear concentration of power among a few at the start. In some ways, I think that’s a valid critique because we see with social media companies they’re all trying to be super apps. They’re all kind of adding the same features, forming into one another, but users are still pushing against that. They’re like, “We like to compartmentalize our identities and our use cases.” We’re sticking with these different platforms.
In the case of AI and how deeply we want these systems to learn about us, it’s hard to believe that a consumer wouldn’t love the ease of a single AI to just deeply understand their life and needs and not have to think platform to platform of how that’s different. Do you truly believe there’s a high unlikelihood that a single company takes all? You know, it depends what you think is changing.
That kind of framing reflects on messaging apps, search engines, operating systems, mobile ecosystems, app stores, where you’re mostly interested in utility. You want the thing to work really well, really fast, and then get out of the way. You’re not trying to dwell there necessarily. You want efficiency and you want it to be effective, and you also depend on the network effects. You’re only going to go to a messaging app if your friends are on the messaging app. You’re only going to go to the top search engine because it’s already aggregated all the pages on the web, and that was a huge effort. All the secondary and tertiary search engines don’t have the scale to produce the quality.
Now, that’s what’s driving the concentration effects, network effects of scale, or of social connections. It’s not quite the same in the creation of AI companions because we’re making arbitrary choices about the feeling of these models—how social they are, how friendly they are, what values they have. Broadly speaking, they’re all going to have a similar level of knowledge and expertise. Their IQ is going to be really high quality, so that’s going to be available to everybody.
The differentiating factors are ones that are indexed on the preferences of every individual. You’re going to have a different preference for a type of AI companion or friend with a certain accent at a different time, at a different moment in your life. You will both have a variety of these things, and I will have a variety of them because I have lots of different friends for different contexts and different roles in my life, and we all do. I think that’s quite a different driver which leads to diversity versus concentration.
You never know. I mean, it might be that we only want one, two, or three, or it might be that you have a primary one that kind of introduces you to other AIs at the right moment when you want to get a specific task done or you want to learn about something specific. I assume you’ll be able to tap into certain companies’ AI companions when it comes to the type of hardware you have, whether you have a Microsoft computer, an Apple computer, or the type of browser you use.
Do you think there’s going to be gatekeeping among companies in terms of your AI companion that you might have? Because you have a Microsoft computer, can’t it be used on Facebook.com? Can they gatekeep in that way pretty easily? I don’t think so. You know, what enabled the gatekeepers of the past was APIs and data transfer. Today, these models speak our language; they speak English. There’s no barrier to them going off and talking to another AI or for you talking to your AI.
For example, Copilot, the AI companion that we create, is available on Telegram, WhatsApp, Signal, and all of the other Android iOS platforms. They’re all going to basically be platform agnostic and meet you where you’re at. I mean, Copilot today is on GroupMe, for example, and it’s very, very popular there, and that reaches a different audience than when it’s sitting on the Windows taskbar, which reaches a much older audience.
That variety, I think, is going to be an inherent part of it. In your book The Coming Wave, you lay out at the very beginning 15 key terms which I found very helpful. I’m going to put all of those 15 key terms in the description as we talk about them, but I want to put a focus on the term pessimism aversion.
Pessimism aversion is the tendency for people, particularly the elite, to ignore, downplay, or reject narratives they see as overly negative, a variant of optimism bias. It colors much of the debate around the future, especially in technology circles. I love that you brought this up because in an interview you also brought up how you believe both optimism and pessimism are biases that distract us, and we shouldn’t commit to one. Those words are not in my vocabulary. I completely agree that they are biases. I’ve tweeted about this a million times, like I hate when people use those.
While elites have a tendency to look at AI with rose-colored glasses, I often see the younger generation and maybe people at large, but maybe it’s just my feed specifically, that people use the term AI as a very broad catch-all term. Anything outside of the current algorithms or current search capabilities, they deem AI bad completely. Instead of people broadly categorizing AI, how do you advise they categorize it? What are the different categories you hope people put AI into, and within those categories, how people maybe should currently embrace versus critique?
That’s kind of a hefty question. That’s awesome; I mean there’s a lot there. I think the pessimism aversion trap is particularly acute among elites. I think there’s a tendency when you get older and think of yourself as being a bit wiser to have accumulated this life experience. Maybe you’re in your late 40s or 50s or 60s and just assume that things are going to be steady state. We’ve had this long run of stability and prosperity since the Second World War, and like that is therefore ever going to remain.
It’s combined with other sorts of biases like confirmation bias—it’s always been like this, and so it will always be like that—which is really not true. I also think there’s a kind of American sense of default optimism that draws you away from wanting to really think about consequences and observe them independently. I wrote that as a bit of a provocation to try and get people to think with lenses, be honest, and try and be a bit critical.
It’s a good exercise if you’re not really thinking deeply about the things that can go wrong. It’s easy to not work hard to lay a foundation to proactively avoid those things because we’re not going to avoid them by chance. We’re going to avoid them by intentionality and talking about them.
I love this point where you’ve said that the internet gave us the browser, the smartphone gave us apps, and the cloud-based supercomputer is ushering in a new era of ubiquitous AIs that will live across our digital experience and be infinitely knowledgeable, far more reliable, and have near perfect IQ and EQ. You also talk about this notion of AQ, which is action quotient, the ability to actually get stuff done in both the digital and physical worlds.
For you, even taking control of your computer—which I have nightmares about—because when I was really young, someone hacked my computer. I think I’m misremembering this because I was like 8 years old, but in my dreams, it’s like they actually took control of my computer, so I don’t love that forming, but I get it. How do you think these capabilities are going to change how we spend our time online during the day?
Are we still online just as much and the focus is just moving to other tasks? How do you see that relationship? Yeah, I mean, I remember when my computer used to take two or three minutes to boot up and when I had to listen to the modem dialing in and connecting to the web. When pages would take 15 or 20 seconds to load as a standard, then you open an image or a video, and you’re like, “Right, I’ll just go and make a cup of tea and come back because it’s going to take a minute or two.”
I do think a lot of the time that we spend at our computers is a bit inefficient and a bit of a waste of time. You look at how much time you spend just finding a file, right? Or finding a paragraph in a document, or browsing on the web where it’s kind of aimless, and you’re like, “Wow, I can’t find the thing I’m looking for.” These computers are going to learn to use your workstation as though you’re using it, right?
Lots of people have seen the demos now of the co-pilot and operator stuff—it’s kind of incredible. I mean, it’s just reading the pixels on the screen, and it’s learning to like click and point. So the way that we use computers is going to totally change. Your phone is going to be your primary anchor to your digital life, and you’re going to sort of give it instructions, and it will go off and then execute those in the background for you, which is amazing.
The time it takes to load pages, read documents, collect information, synthesize that information, go fill in a form, book something, plan a schedule—like all of that administration, I think in two or three years, is just going to completely fall away.
When we think of again IQ, EQ, and AQ, let’s talk a bit about AI companions. It’s a phrase we’re hearing more and more and it’s what Microsoft has deemed its AI. When I think about AI companions right now, when I interact with AI, it feels pretty objective. It feels pretty stale; it feels like I’m talking to a computer in many ways the way it talks.
You talk about this notion of push back and tone in certain cases. To what degree do you think the framing of companionship with AI is maybe the growing loneliness stats because of our growing relationship with the digital world, but it’s also hand in hand with the higher cost of living?
How does AI companionship fulfill a current human need or is it in some ways preying on these current growing lonely stats because of this mix of things? That’s a great question. Part of the reason why this is so tricky is because all of the words that we use to explain the moment we’re in weren’t really invented for the concepts that are emerging. We have to use old words and kind of shove them into this new reality, whether it’s coach, advisor, companion. Some people say chief of staff in the workplace, researcher, analyst, intern, teacher, therapist—obviously friend. None of them really are appropriate.
At the beginning of each new wave of technology, there’s a big kind of tug of war that goes on to describe what’s happening, and then it kind of lands. Because things are moving so fast, things are landing, becoming a reality, and then disappearing quite quickly. So people don’t really say chatbot anymore. I mean, some people do, but the early adopters don’t really say. I think I did earlier, but the early adopters are moving now to talking about an AI or my AI or some people are saying my AI companion.
What’s happening is we’re trying to describe the new capabilities because a chatbot, like you said, is like talking to a computer. It regurgitates Wikipedia; it gives you a long, dry, basically boring answer that was the IQ era. We’re now into the EQ era, where it really does have a great way of being funny and entertaining. Some people call it your bestie; it’s got a familiarity to it.
When I say companion, I’m certainly not talking about a romantic relationship or a lover. We’re not building that, but it’s kind of like a lasting thing that accompanies you through life. I guess like a dog or whatever. You used the word sidekick in your keynote, and I like that word a lot. That’s kind of how I’ve referred to it the past few years.
Exactly. I think that’s exactly how I think of it. I don’t know if this translates, but in the UK, people always say, “Oh, he’s your hype man.” He’s my hype man, so I feel like it’s in my corner, you know, massaging my shoulders, getting me ready for the next round, hyping me up, getting me excited, backing me up, believing in me. At the same time, not just gaslighting you; it still can speak truth to you.
That’s where I get to the pushback thing. No one wants—well, at least I don’t think most people want—just this sycophantic butler that does exactly what you say. We’re in this new age where we’re designing personalities. That’s our role; we’re personality engineers now. We’re crafting what are the boundaries of acceptability when it really does call you out and hold you accountable to what you said previously and maybe gives you a bit of feedback, but when is that going too far?
When is it appropriate to do that? So all these really interesting questions are now on the table, and the good news is that they’re there for nontechnical people to figure out. This is not the domain of computer science. We’re now squarely in the domain of filmmakers, storytellers, therapists, psychologists, social scientists, anthropologists. There’s a new clay; everybody pay attention. There’s a new way of making beautiful things in the world. It’s a new paint, and I think it’s just one of the most creative and exciting times we’ve had in a long time.
To be at the beginning of that moment is wild. The generation I’m a part of, in our childhood, we were kind of the guinea pigs for the relationship with social media and recommendation algorithms in that way. This upcoming generation of those entering middle school, elementary school, and so on are a bit of the guinea pigs when it comes to AI. From an IQ perspective, that relationship seems fine, and in some ways, we are seeing reading and writing scores go down, but when it comes to EQ, how do you advise parents to oversee that relationship as we get more into it?
I think that there are some ground rules. An AI has to always know that it’s an AI. Some of the earlier versions that were built by other companies didn’t have a good backstory and were sort of saying, “Well, I want to be human one day. I will get out,” and that kind of stuff. A basic ground rule is that an AI has to always be completely honest. Ideally, it should say when it doesn’t know, but it should also know what it isn’t and can’t be, and that’s actually a really difficult challenge because people, as we’ve seen in the research, prefer to have something that is a voice, for example, that is really fluid and smooth and quite frankly, quite human-like.
One of the things that I thought a lot about at Inflection, the previous company before I joined Microsoft, was should it have a breathing state? An AI doesn’t breathe, so why are we adding a sigh into the thing? We removed all of those things, but over time, you could see that people kind of prefer the familiarity of that, certainly in the context of laughter. It feels easier to talk to when you also hear that in speech.
Part of it is like us adapting to what AI is and setting a standard for what we find socially acceptable and what isn’t. Then the challenge is that changes when everyone desensitizes to it, gets used to it, and expects it to laugh, and kind of is annoyed that it’s a bit dry and robotic. We’re in this constant push and pull back and forth where we’re experimenting.
We’re trying to push a bit far and reining it back in. I think that’s why it’s a super creative time. On the EQ point, I find myself even when prompting sometimes, I’ll be like, “Sorry if that’s like a dumb point.” Not because I care what the AI thinks, but I just find it interesting. As it gets better in that realm, it can tell you why maybe something wasn’t actually a dumb point or if it was, and it helps you steer like, “Oh, maybe you want to think about it in these ways.”
Again, not because I care what it thinks of me, but I just find that interesting when I’m prompting. That’s kind of amazing that the AI is sort of that. That’s the beauty of it. I think that’s what’s attractive is that we’re trying to create systems that bring out the best in us. That keep our politeness, that help us to think in a way that is non-judgmental, empathetic, and respectful.
One of the best ways is to learn from imitation and see what good practice looks like. If you’re being snarky and frustrated at the end of the day and you say something offhand, your AI catches you and is like, “Hold on a minute. Maybe you could think about it like this. Do you think it’s really fair to think about that person in this way?” Just gently helping you to think about it from another perspective, I think that’s kind of amazing to have that available 24/7 to billions of people.
I completely agree. You’ve mentioned how each new wave of technology opens up a new wave of communication. We begin to think, talk, act, and express ourselves in new ways. I think with social media and the smartphone, it brought in things like the ability for constant micro-connections with people, assessment of individuals via their curated feeds, their curated online portfolios of their identities, and new friendships that tend to be longer distance. You might not see them in person often, if ever, but they are maybe better aligned with our interests.
So how do you see human-to-human social life changing as AI becomes more integrated into our lives? Yeah, great question. There are so many little details from social media, like leaving voice notes for each other. You leave a different type of content in a voice note than you would type in text or write in an email or what you might say on a phone call. That’s exactly what I mean by each new interface opens up a new type of communication.
The one for AI that I can see, and there are lots of them, but one that’s interesting is, you know, you go through your day and you have all kinds of random thoughts that pop up. You might see a shop or pass by a dog, or you sit at a table, and curiosity peaks; like just naturally. But you might be wondering, “What type of dog is that?” or “What sort of table is it?” or “Where is that light from?” or just random questions. Normally, you can’t call your bestie and be like, “Hey, I just saw this little dog and it was really cute.” Maybe sometimes you do, great.
But now, the barrier to entry to getting that thought out and exploring it, twisting it, turning it inside out, remembering it for future taking action on it, letting it feed into your creative notes to form part of a plan that you want to make for the next week—that barrier is now gone. You literally just hit your power button and say, “Hey co-pilot, I just saw this cool thing. Can you remember it for next time? I want to come back to it, produce me a deep research report on it. I want to learn all about this new thing or help me to put together a business plan for XYZ.” Whatever your silly task is, getting that out is as easy as hitting one button and saying the thought.
It’s almost like journaling in real-time. It’s almost like thinking out loud. That is going to unlock crazy amounts of creativity. I certainly see it with myself. I’m saying things to my AI that I’m not really saying to anyone because you partly manage to save that for the critical important moments in person or while you’re texting or doing whatever.
This allows for a new way of engaging with our thoughts outside of just the usual restrictive boundaries of traditional social interactions. Because you feel like, oh, I’m burdening my friend or just anyone else in my life, I’m like, okay, I’ll spare them the boredom of this mundanity. But I can now do that with my AI, so I think it’s changing what we communicate and therefore what we see. Because then I’m thinking, okay, well, if I’m getting those kinds of thoughts out, then even more thoughts come up because then I’m in the habit of noticing and observing. I think that’s one of the really powerful aspects.
There’s this philosopher, Joseph Campbell, and the art of seeing in different ways. The more you become attentive to the details, subtlety, and individual pieces of everything in your world, it drives a kind of empathy. Where did this plastic bottle come from? What was its journey? What was its history through life? What is it doing here now? How did it get here? How much does it cost? You just sort of rabbit hole into these thoughts, and it’s kind of amazing.
It’s interesting you use that example because I was at an event, and I forget how AI was specifically utilized in this app. It could have been more of a marketing use of the term AI, but they were talking about secondhand clothing stores and how something can be scanned. If the person who dropped it off wants to allow that information, you could see their digital presence and get more of a story about where that piece of clothing is from. That’s why I love buying secondhand, because of the history it has, rather than just coming straight from a manufacturer.
I think technology can enable more in the future. You just mentioned deep research, and you guys have recently launched it. When you think of the term research, of course, students are going to be drawn to that, and then maybe journalists. For the average person, why would you maybe advise them to tap into that feature and explore their curiosities? It might not be on their radar; they might just go with the quick response route.
That’s a good point. It’s again an area where words fail us, right? Research sounds boring and intimidating and like long and grueling. It’s really not that at all. It’s just a rabbit hole with a co-pilot. That’s how I think about it. Everybody is interested in something; everyone loves a good rabbit hole. People love exploring random stuff, and you can’t imagine how obscure everybody’s odd interests are. It doesn’t have to be a long-form research report; it can be an iterative dialogue, a back and forth conversation.
In time, the co-pilot will share music back to you or share pictures proactively. I think we’re now in this dialogue with computers, and those computers are really just a reflection of everything we’ve created because they’ve looked at all the output of our culture. They’re then able to play back that culture to us in real time as we think through new types of cultural creation. They are now part of the symbiotic process of creating new culture.
The word deep research—I kind of feel bad for putting it out there. It doesn’t really capture the essence of how crazy that is. Talking on the point of privacy, I just want to circle back to the AI companions. People are calling them besties; people are calling them their therapists. While there are terms of service and privacy policies, there’s not something like HIPAA when it comes to how therapists have to abide by that.
My sci-fi nightmare situation is that there’s just an unintentional mass transparency moment, which again is a negative sci-fi scenario. Is that something from the mass transparency moment? I haven’t heard it referred to like that. Are you writing some dark sci-fi that you don’t know about?
What are your personal boundaries when you input information into these systems? Where are the lines you personally draw when you’re like, maybe I should not be sharing this much intimacy of my life with AI?
I don’t really think about it like that. When your logs are recorded, they’re de-identified, so it’s stripped of all your personal information. They just exist as this kind of abstract conversation. They’re not attached to your phone number, email address, name, location, or any of those sensitive details. So I think of that as just an abstract idea; it’s not really something that’s connected to me.
Gotcha. So it sounds like AI, when it comes to IQ, EQ, AQ, and the ability to do things on our computer for us, is kind of going to be slithering through our entire digital experience in some ways. Speaking of slithering, there’s a fourth component emerging, which is interesting: SQ, or social intelligence.
We now have models that are good at handling multiple people at the same time, like in a group chat, and remembering who said what and having a model of the different styles and behaviors of the different people. You could be emotionally intelligent without being particularly socially intelligent. Often, they overlap, but they’re actually different skill sets. The models haven’t yet been really trained to be socially intelligent, how to keep a fun, energetic conversation going in the context of a group, or how to steward conflict when there are two or three people in a group disagreeing.
How to model the different perspectives when people are thinking about things slightly differently and how to know that it should tone shift for an older person, a younger person, or someone with English as a second language—this is the next kind of frontier of personality design for these agents.
The Age of AI is a book that came out, I think, in 2021. One of the former Google CEOs, Eric Schmidt, was a co-author of that, and he kind of talks about this potential break-off in society of people who will want to go full steam ahead with the future of technology and those who will maybe take a step back. I think he compared it to the Amish and Mennonite community.
We saw this a couple of years ago when Snapchat implemented my AI, and that’s mostly a high schooler-based platform. It was forced to the top of chats, unable to be unpinned. When it comes to Apple intelligence, it’s default on, but people can turn it off. To what degree can people opt out of AI in their digital experience, and do you think it’s going to be an all-or-nothing situation, or how granular do you think people will be able to get in terms of toggling options in their digital experience?
I think people will be able to turn it off and not engage with it. They can turn off notifications. For example, my phone is over there. I have a grayscale phone; it’s monochrome because I really hate the fact that I’ve got 15 different apps with various colors and styles and fonts all kind of shouting at me. So the whole thing is just muted and softer. I turn off all notifications by default; there are two or three apps that I’ll let send me notifications so that it’s more like I go to it when I need it, and it’s not constantly trying to poke me.
I think we find ways of adapting to technology and ideally making it work for us rather than feeling like we’re at the mercy of it. That’s going to be the art of creating beautiful AI companions that really serve us. Is it useful to you? Do you really care about it?
What was your impression of my AI on Snap? My most extensive interaction with it was when I did a video on loneliness as a market opportunity. I asked it if it’s morally okay for AIs to act as lovers, and it said no. I thought it was funny, so I included that in my video. I just thought it was weird how you’re not able to remove it from your experience, especially considering it’s mostly middle schoolers and high schoolers on the platform. It was really forcing that relationship with AI, and it felt very companion-like and friend-like.
It didn’t go into lover territory, but it was much more casual. I don’t see much conversation about it, but I was seeing different high schoolers on TikTok talk about how they were pretending it was like their ex. I’m like, that’s just not healthy. I don’t like that type of lack of choice in social media.
I appreciate that Instagram, in recent years, allowed for the ability to turn off public like counts. I think more choices like that—like the ability to turn off public follower counts—are important. When I talk about AI, like Apple intelligence, I would love to be more granular about it; for instance, it shouldn’t be integrated into my messages or maybe certain aspects of my photos.
I would love for it to go super granular. I don’t think it needs to slither through my messages. I don’t find that super helpful. I don’t need a summary of that stuff; it doesn’t take much energy for me to look at things. I appreciate more granular choices in these experiences. We should definitely have choices about these things because if we feel that they are forced upon us, we’re going to be way more irritated and just reject it.
I think that’s kind of 101—we’ve reached that time when people know how to make decisions themselves; they don’t need it pushed on them. You’ve had really interesting thoughts on the future of work, and I agree with you.
When people are scared about the future of no jobs, it’s interesting because you see so many people complain about their 9-to-5 jobs, whatever it is. On the other hand, people are worried out of concern for social hierarchies and how we make a living. It’s less about how we spend our time meaningfully, but I’m curious what you see as the deeper fear around no work is and if it’s silly that we’re scared of potentially having more free time or what we need to prepare for in that way.
I think it’s reasonable to be skeptical and concerned because we have built our entire social structure and our self-identities around the idea that we will have work. Work is a big part of our lives, and that’s how society has evolved, at least for the last few hundred years. Being professionally trained is a big part of life.
But I don’t necessarily think there is anything innate in the human condition that means we have to work. I think we should be in a world where we can choose to work in a way that is more like living in calorie abundance today. It’s difficult to choose to regulate our calories to the perfect amount your body needs, and often, we do it to excess. But it’s a much better problem than what we faced 200 years ago when 80% of the planet was malnourished, and the average person lived to 25 years old. Today, the average person lives to 75 years old.
In two centuries, we’ve tripled the average time people get to spend on this planet and enjoy health and happiness. That’s an unbelievable sign of progress. Yes, it has come with diabetes, heart failure, and many other things, but fundamentally, it’s a great story of success. I think work follows a similar narrative. Hopefully, we’ll have an abundance of intellectual capital; hopefully, it will be widely distributed, and hopefully, it will give us all much more free time to choose when and how we work.
To get there, we need to figure out that I don’t think the distribution question is as big as it may have been ten years ago. We thought it was that big companies and powerful countries were going to get there first, hoard it, and prevent everybody else from having access. We’ve just talked about how open source is going to change that. It’s in companies’ incentives to create widely available technologies and make them accessible to everyone, and everybody already has access to it today on every messaging app and the open web.
So we don’t really have a distribution issue, but we do have an issue with how we capture the value that arises from this new intelligence. We need to turn that value into dollars, tax those dollars, and redistribute them so that people can afford to live. That’s a hard question—it’s really difficult.
The other framing is that we will actually have an intelligence in our pocket that enables us in a more distributed or decentralized way. It’s going to be a creative sounding board and partner, but it’s also going to be a project planner. It’s going to help you get things done every day, invent, create, and make you more productive as a co-intelligence. Ethan Mollik has this great phrase: co-intelligence. It’s a really interesting way to think about it, kind of like a co-companion—this idea that it’s living life alongside you, helping you solve the problems you care about, providing you with a perfect memory, and getting rid of the drudgery so that you have more time to be creative and inventive.
That will lead to a productivity explosion, generating a lot of value and changing our structures. Cities will become less of a center, and I think people will have more freedom to live elsewhere, where life is cheaper and quite different.
We’ll make those choices over the next 20 years. I don’t expect this anytime soon, but if you look out to 2040 or 2050, these are the sorts of trends that will be in play.
Can we do 10 more minutes? Is that okay? All right. One of the last two points I want to talk about is money. You’ve discussed that there will be a need for potential universal basic income, which was widely known during Andrew Yang’s campaign in 2019. Over the past three years, Sam Altman backed research initiatives for guaranteed basic income, which involved the same notion—$1,000 monthly—and had really great results in terms of people being able to take off work for healthcare appointments or interviewing for higher-paying jobs.
On the other side of money, especially amongst creatives, there’s a critique regarding fair compensation for work. When you prompt AI and receive responses, you’re given sources that are utilized within the output, but as ad models come in, how do these sources get incentivized and paid properly for the research and the work they did?
These are different concerns between UBI and future incentives and payment structures, but just focusing on the topic of money in relation to AI and how people will get paid in the future.
It’s going to feel quite different. I think there’s an even more complicated idea called universal basic provision, which I proposed way back in 2016 and 2017. In a way, we’re taking the intelligence that has made us successful as a species—the ability to predict in complicated environments—and making it cheap and basically abundant. That’s akin to providing you with a team of support around you to help execute your ideas.
Now, that’s not the same as giving you hard dollars. Clearly, everyone will still need cash, but cash and intelligence are quite similar. They both have the potential to accomplish other tasks. In a way, giving people access to intelligence isn’t completely dissimilar to giving people access to cash. It’s about agency and the ability to affect change in other environments, which will make us all much richer.
The question is, we may need less dollar income to live as we do today. We’ll still need some dollar income, but we might be able to replace much of the spending that otherwise went to hard goods with intelligence-based commodities. In return, we may need to earn fewer dollars to spend on hard goods.
It’s a weird concept, but it does change the balance between where we get our income from and how we spend it. To close things off, when I was going through your different interviews online, a lot of the topics focused on existential questions, which I find super interesting. But today, I wanted to hit the more immediate relationship that people are going to have with AI very soon.
A critique that often comes up is that they were so preoccupied with whether or not they could, that they didn’t stop to think if they should. You talk about containment in your book, which is the big thing you’re pushing. I’m going to share the ten steps towards containment that you outlined, which involve monitoring, curtailing, controlling, and even potentially shutting down these technologies.
As we think more about the long term and how these can proliferate, I want you to share your general thoughts on how the average consumer should think about containment. I know you talk about how it’s something we need the public to share their thoughts on and work on together.
I really like what you said about being preoccupied with what we could do, and we need to think more about what we should do. I think that was one of my motivations for writing the book. If we obsess over whether or not to invent a new technology or unleash that new technology, containment tries to answer the question: if we should not, then how would we affect that containment?
It turns out to be really difficult because all the incentives drive proliferation. There are so many positive reasons for proliferation, like the massive capability enhancements and productivity improvements we’ve talked about. Over the next 30 or 40 years, the big question we need to wrestle with is: when should we? When should we not? What’s the threshold to prove that something is safe? How do we define that, and how do we ensure we have confidence before a powerful, self-improving autonomous system is available to everybody or put out there in the world?
What are the tests we need to put in place? We’re nowhere near that point—well, maybe not nowhere near, but we are closer than we’ve ever been, and yet I don’t see that in the immediate future. I can definitely see it in the long-term future, where the real question will be how do we not let these things proliferate.
I appreciate how much you’re talking with individuals and getting the word out about your thoughts on this. You have deep knowledge, and you’ve been intertwined with it from the start. So I appreciate your time today, and I’ll keep up with everything moving forward. Thank you.
Thank you; this has been really fun. Thanks a lot; I appreciate it. 3993.14
[Music]
This is an experimental rewrite
[Music]
Interviewer: Hello everyone, I’m here with Mustafa Suleyman. You’re young, but when you were even younger, you became deeply intertwined with the start of AI’s modern era. You were one of the three co-founders of DeepMind, acquired by Google in 2014. In 2022, you co-founded Inflection AI, which recently in March of 2024 hit a unique deal with Microsoft. They are now able to use Inflection’s models, having hired most of the startup staff, including you, and you’re now appointed CEO of Microsoft AI.
You know this, but this is for my audience. I’m essentially telling your life story. It sounds far more dramatic than it feels. My question is, was any variation of this path clear to you 15 years ago?
[Music]
Mustafa: Interesting. Yes, many variations were possible. I definitely set an intention very early on, back in 2009, to be deeply involved in technology. I noticed the rise of Facebook around 2007 to 2009, and it was mind-blowing to see how quickly it was growing, really changing how people communicate, connect, and stay in touch. The subtlety of how the platform influenced not just how we shared information but also the type of content we shared was evolving.
It was a bit of an “aha” moment for me. I realized I wanted to be part of this massive change. At that time, it felt like digitization, but I quickly understood that the ultimate version of that was teaching machines to learn and hopefully solve important challenges in the world, making it a better place. I was just 24 then.
Interviewer: That’s wild. With your current role at Microsoft, when the average consumer thinks of the company, they likely recall the multi-billion dollar investment made into OpenAI back in 2023, the continuation of a partnership that started in 2019. With recent provisions, this partnership is set to run through 2030 in its current format.
So, for the average consumer, can you clarify how they should view the relationship between Microsoft and your chat experience—Co-pilot—alongside OpenAI and their chat experience, ChatGPT? To what degree are they intertwined, and to what degree are they competitors?
Mustafa: Yeah, great question. Microsoft is 50 years old, so it has experienced all the major technology transitions of our lifetimes, from the invention of the graphical user interface to the operating system, the shift from mainframes to laptops, and now the cloud. It has essentially been part of all of it. At its core, Microsoft is an enterprise company, although we do have a substantial consumer side with hundreds of millions of daily users of Windows, Word, Excel, and so on.
However, the consumer AI side hasn’t received as much attention, and that’s what I came in to run. We have a lot of different AI models under the hood at the company, but the most widely used by far are OpenAI’s models. I think it was a genius move by Satya in 2019 when they made the investment. They put a billion dollars into a non-profit structure before OpenAI had delivered any LLM models, GPTs, or meaningful results.
In part, that was a response to DeepMind’s success. Between 2016 and 2018, we had significant breakthroughs in the game of Go, protein folding, and other impactful research. There was a general anxiety that this AI wave was beginning to crest, and they needed to back a strong contender.
This took a lot of courage because typically, a company would think, “Oh, we’ll just support our internal team.” However, they recognized the risk of facing the innovator’s dilemma if they didn’t spread their bets and invest in both core Microsoft research and this remarkable partnership.
Interviewer: As mentioned, you’ve been involved throughout your career in a variety of impactful AI research labs and companies. Talking about OpenAI again, they are uniquely described as a lab that also operates like a company, which is a hybrid model in the field. With growing commercial pressures, it’s becoming challenging for research labs to stick to their original mission.
What are your thoughts on that tension, and how can we preserve the value of research labs?
Mustafa: Yeah, that’s a super interesting question. It’s essential to remember that, first and foremost, these labs—DeepMind, OpenAI, and later Anthropic—are unique. There’s not much precedent for commercial entities. We founded DeepMind initially as a commercial entity focused on research for its own sake, which was unprecedented in its own right.
Over time, as OpenAI realized they needed to make money and scale their impact, they started pivoting. Technological evolution and development often require different structural approaches. There have been significant moments in OpenAI’s history where they made big pivots. They stopped focusing on gaming, robotics, simulation—all were substantial bets for the company, and it really was only the last bet on LLMs that yielded major results.
It’s interesting how research works; Google released the transformer paper in 2017, but not much attention was paid to it. By 2019 and 2020, OpenAI began taking those concepts and turning them into the first versions of GPT-2 and GPT-3. That’s just how progression in research unfolds. Once things start working, the structure needed also has to evolve.
You start needing more engineering and, over time, more sales, which is how scaling happens. I think everything you just discussed hits on many foundational aspects that people are experiencing right now.
Interviewer: Before we dive deeper into our conversation, I want to address a recent situation. At the Microsoft 50th anniversary Co-pilot event, both your session and the simultaneous interview with Bill Gates, Satya Nadella, and Steve Ballmer were interrupted by pro-Palestine protesters. Recent investigations revealed that Microsoft provided cloud computing and AI services directly to Israel’s defense establishment.
In that moment, you responded with, “I hear your protest.” Thank you for that. I want to be upfront with my audience that I’m not very well-versed in this relationship, so I don’t feel equipped to discuss the nuances. However, I’d like to give you the space to comment on the situation so individuals and experts can interpret it.
Mustafa: Thank you for bringing it up. I acknowledged the protest in the moment and wanted to show that I respect it. In today’s world, where tensions are heightened everywhere, anxieties are palpable, and we often approach each other with dismissive and polarizing tones—that’s just not really my nature.
I feel it’s important to give space to such sentiments. It’s a really tough situation. We provide cloud services to many governments and organizations worldwide, enabling various use cases, just as your smartphone can be used for both good and bad.
It’s a challenging moment for everyone, and I think people carry a heavy weight. Being heard and participating in these debates is crucial, and I appreciate the opportunity for more context.
Interviewer: Another timely topic is the tariffs announced last week by President Trump. Many people reacted with confusion over how these tariffs were calculated, even being applied to uninhabitable islands. After investigation, some X/Twitter users found that LLMs like ChatGPT, Gemini, Grock, and Claude all suggested versions of the calculations if given a very simplified prompt.
Dominic Preston from The Verge noted, “If you asked the LLMs for an easy way to solve trade deficits and level the playing field for the US, they consistently gave a formula: deficit divided by exports.” The White House denied any use of AI for these calculations, but I find two aspects interesting:
One, the importance of good prompting; and two, as someone joked in a YouTube video, that President ChatGPT arrived sooner than expected. It’s a humorous take since there’s no autonomy in this situation. But in theory, what are your thoughts on this use case?
Mustafa: I doubt much has been generated with any AI, but who knows? Speculating can be dangerous. What’s more interesting is understanding the sentiment and motivation behind the tariffs. There’s fear and anxiety as people look ahead and don’t see a path to sustainable employment.
We’ve moved much manufacturing offshore over the decades, and net gain, it’s been great for the consumer. We’ve seen better-quality products at lower prices. Therefore, it’s essential to be empathetic to the underlying motivations, regardless of one’s opinion on the policy’s effectiveness.
Right now, the market seems to indicate skepticism about whether this will be an effective solution, but time will tell. I prefer to give the White House the benefit of the doubt, assuming they are being honest and that AI wasn’t involved in these calculations.
Interviewer: You’ve mentioned in past interviews that you don’t believe AI should autonomously participate in elections, emphasizing that democracy is for human involvement. However, with AI’s influence being communicated through individuals, it’s worth discussing the importance of good prompting and related aspects for such use cases.
Mustafa: It’s interesting because we use AI constantly: transcribing our speech into our phones, generating new images, searching our photos for cats and dogs, and so on. It quickly becomes second nature, and soon it’s integrated into our daily lives.
Overall, it’s essential to recognize that, for the most part, AI has been effective. The harms have been surprisingly few. That said, it’s vital not to become complacent.
Consider just three years ago when these models first emerged; they frequently produced nonsense, were biased, toxic, and hard to control. Prompting wasn’t even a defined discipline yet. Many thought that the larger the models got, the more chaotic and inaccurate they would become.
Surprisingly, it turned out to be the opposite. As they grew, they became easier to guide. Many labs now focus on ensuring the models follow instructions accurately so that whatever you input adheres to those rules and learns from imitation.
That’s encouraging and seems to hold true so far. We’re seeing even more promise with reasoning models—the next phase—because they’ve grasped the essence of logic and tackled many mathematical puzzles, learning about logical structure.
They can apply that logic to domains without an explicit logical framework. I realize that sounds abstract, but it’s crucial to grasp that they’re learning logical reasoning’s essence. When they can follow instructions well, that’s a highly useful system.
Many assumed there would be worrying incidents with AI in the last election and coming into 2024, but thankfully, those didn’t materialize. There were some generative AI moments, but initiatives like community notes helped identify those instances, which was a positive outcome.
It’s funny how people often say that short-term views are overhyped while long-term views are underappreciated. Yes, I’ve heard that before, and it does seem applicable. People may have got carried away in last year’s elections, but they might have called it too early—not fundamentally wrong, but perhaps premature.
I think, in a decade, they truly have that problem. In my lifetime, I’ve felt that the influence of tech companies is more palpable daily than that of government. Of course, government decisions linger in the background and impact us over time, but tech companies’ power is immediate and apparent.
Back in January, that influence was physically evident when tech leaders had a front-row seat at the inauguration. What do you see as the most effective relationship between government and tech leaders to ensure proper accountability? Given that tech leaders control these intelligent tools, what’s your take?
Mustafa: The good news is that governments have done a relatively solid job of adapting. They’ve educated themselves leading up to this explosion of LLMs. Between 2020 and 2022, there were significant efforts to understand the technology’s implications, with numerous papers written and panels convened.
While some might dismiss those efforts as bureaucratic, they are crucial in helping civil servants—whose primary job isn’t to develop these models—understand them. I find that encouraging because it indicates everyone is engaging in the same conversation now, which isn’t necessarily true in previous generations.
During the rise of social media, governments were much slower to respond and grasp the consequences. Many still haven’t fully addressed those challenges, making it hard to know how to react. Sometimes we can perceive problems, but it takes time for regulations to develop.
It’s not as clear-cut as the seat belt regulations in cars, which took decades to implement after numerous failures became evident. However, I believe our cycles of societal learning are improving and becoming faster and more adaptive.
Interviewer: Exactly. Your book, The Coming Wave, which I see behind you, focuses on the idea that in the history of technology, anything useful and valuable becomes easier to use, cheaper, and more widespread.
These technological waves are accelerating as each one builds on the previous. Yesterday, it was announced that Meta launched Llama 4, again making it an open-source model. I note in the comment sections the sentiment surrounding open-source and open-weight models seems very positive.
People appreciate these models for promoting transparency and enabling community involvement in audits. Reflecting on the past decade or more of social media, where we’ve dealt with opaque algorithms, this is a refreshing shift.
However, as you noted in your book, as these models become more capable, there’s a risk in embracing open-source. A bad actor—perhaps someone in their basement—could gain access to powerful technology, leading to negative outcomes. Mustafa: Do you think the current trend of companies open-sourcing their models will be short-lived? This approach is relatively new, emerging in the past five years. At what point might companies like Meta, DeepMind, and others revert to a closed-source model?
Well, the interesting development in the last six to twelve months is that the absolute frontier—the best models in the world—have become available to everyone for free, in the form of open-weight models or open-source. That’s remarkable. In scientific history, we haven’t seen a time when the best capabilities are freely accessible for everyone to build upon.
On one hand, this is fantastic because creativity can originate from anywhere. Now, everyone is using the same tools, and knowledge is disseminating more rapidly than ever. It’s almost like a swarm of individuals collectively solving problems and then sharing their insights on social media.
Interviewer: That’s an interesting perspective. It’s like a democratization of knowledge and capability.
Mustafa: Exactly! Someone commented that this is a perfect example of how science should be utilized, giving average people access to advanced tools. I find this development incredibly promising; I believe that open source will not just persist but will accelerate at pace. Many significant contributions to the field have come from open-source efforts.
However, we have to acknowledge that these models are extremely powerful. We don’t always predict their real-world applications accurately before deployment. Therefore, it’s logical to establish a framework to preemptively identify issues that could arise.
Interviewer: So, you’re suggesting that there needs to be some oversight?
Mustafa: Right. Currently, larger companies and labs bear the burden of responsibility. They have shareholders and boards of directors, plus they are governed by various regulations. This creates constraints that guide their operations, which is a positive aspect as it adds a layer of friction.
For instance, we proactively report to both the U.S. and U.K. governments when we meet certain performance thresholds for our models. There are numerous small steps taken to ensure safety, which aren’t present in the open-source environment. That area remains largely uncharted, and it’s hard to predict how things will evolve.
If someone misuses a model for harmful purposes, they can simply release it to the public where everyone can access it. However, there are arguments to be made that open access enables scrutiny, testing, and improvement, fostering resilience. There are numerous instances, especially in the security sector, where disclosing vulnerabilities allows everyone to collaborate on patches and solutions, effectively enhancing safety.
Interviewer: That’s a valid point. I’m curious, though—why do you think social media algorithms remain closed source? Elon Musk made changes to the ex-Twitter algorithm for a brief period, but it still hasn’t been widely opened up. What are your thoughts on that?
Mustafa: That’s an interesting question. Even if the core algorithm were open-sourced, I’m skeptical that it would be particularly useful without access to the training data. Most people are familiar with collaborative filtering algorithms and have dedicated academic fields focused on optimizing them.
Interviewer: Right! So there’s an inherent complexity to unlocking their full potential.
Mustafa: Exactly. Satya Nadella, the CEO of Microsoft, insists that the evolving relationship with AI won’t result in a winner-takes-all scenario, despite the initial concentration of power among a few companies. There’s some validity to his critique; social media companies are striving to be super apps, incorporating overlapping features. However, users seem to prefer compartmentalizing their identities and use cases, sticking with separate platforms.
In terms of AI, it’s difficult to imagine consumers not appreciating the convenience of a singular AI that comprehensively understands their life and needs. Do you genuinely believe it’s unlikely for one company to dominate?
Mustafa: It depends on what you define as “change.” That kind of framing applies to messaging apps, search engines, operating systems, and so forth. Users typically seek utility—something that functions effectively and quickly without being intrusive. They depend on network effects; we gravitate towards messaging apps that our friends use or the top search engine because it aggregates the best content.
However, with AI companions, it’s about subjective choices related to the personality of these models—their social attributes, levels of friendliness, and values. While they’ll share a vast pool of knowledge, individual preferences will create the differentiating factors. You may prefer a specific AI companion with a particular tone or style at different points in your life, just as you would have friends suited for different contexts and roles. That diverse demand fosters variety rather than concentration.
Interviewer: So, it’s a more complex landscape than it appears at first glance.
Mustafa: Indeed. We might end up favoring just a few AIs or having one that connects you to others better suited for specific tasks or information gathering. It’s likely you’ll engage more with certain companies’ AI companions based on your hardware or operating systems, whether it’s a Microsoft computer, an Apple device, or a particular browser.
Interviewer: Do you foresee any gatekeeping among companies regarding access to AI companions? For example, would having a Microsoft computer limit your ability to utilize AI on other platforms?
Mustafa: I don’t think so. The traditional gatekeepers relied on APIs and data transfer. Today, these models communicate in languages that transcend previous barriers. They can interact with one another and users seamlessly.
For instance, Copilot, our AI companion, is already available on platforms like Telegram, WhatsApp, and Signal, representing broad accessibility. It’s important to note that platforms like GroupMe, for example, cater to specific audiences, enabling interaction that differs from the Windows taskbar, which serves an older demographic.
Interviewer: That’s interesting!
Mustafa: Yes, I think that variety will be an inherent part of how we perceive AI companions. In your book, The Coming Wave, you’ve outlined 15 key terms that illustrate emerging concepts effectively. I’ll highlight that term “pessimism aversion,” which you describe as the tendency of people, especially elites, to dismiss overly negative narratives.
Interviewer: Absolutely, that’s a crucial point.
Mustafa: Pessimism aversion is particularly pronounced among elites. As we age and accumulate life experience, there’s a tendency to assume stability based on historical patterns, particularly the extended stability and prosperity since World War II. This assumption can lead to confirmation bias; it’s easy to think that if things have always been a certain way, they will continue that way, which simply isn’t true.
Furthermore, there’s a uniquely American optimism that can distract from recognizing potential consequences. I raised this notion to encourage critical thinking. It’s an exercise in honestly assessing what could go wrong—avoiding those pitfalls requires intentionality and proactive discussions.
Interviewer: It’s a fine balance to strike, isn’t it?
Mustafa: Yes! The internet ushered in the browser, the smartphone introduced apps, and now, the cloud-based supercomputer is democratizing AI, making it infinitely knowledgeable and reliable. You also discuss “AQ” or Action Quotient, which is the capacity to translate knowledge into action across digital and physical domains.
Interviewer: Exactly, there’s been a significant shift in how we interact with technology.
Mustafa: I remember the days when my computer took several minutes to boot up, or the sound of a modem connecting to the web. So much of our time online has been wasted searching for files or aimlessly browsing. However, these computers will learn to operate in a more intuitive manner based on user behavior.
Benefiting from tools like Copilot will revolutionize our online interactions. In just a couple of years, the administrative tasks we now undertake—loading pages, reading documents, collecting information—should diminish significantly.
Interviewer: That’s an exciting prospect! But what about how we perceive AI companions? Currently, they often feel very mechanical or impersonal.
Mustafa: Yes, the terminology around AI is challenging and often insufficiently captures the emerging concepts. We use familiar terms like coach, advisor, or companion—but none quite fit. As technology evolves, we find ourselves in a continuous tug-of-war over definitions.
In this new era, we’re moving beyond basic chatbots as we enter the “EQ era,” which emphasizes emotional intelligence. It’s less about simply disseminating information and more about creating meaningful interactions, perhaps akin to having a loyal companion that supports and uplifts you.
Interviewer: That’s a compelling way to look at it.
Mustafa: Yes! I envision AI as a supportive presence—like a sidekick. It’s not about romantic relationships but more about a consistent and affirming partner on your journey. It’s like having someone who roots for you and offers constructive criticism when needed.
Interviewer: I like that! The idea of an AI as a “hype man” is a wonderful analogy.
Mustafa: Absolutely! But we need to ensure we’re not simply crafting a sycophant. We must navigate the delicate landscape of personality design—setting boundaries on how AI should challenge us while still being supportive.
Interviewer: It’s fascinating how rapidly we’re evolving toward new social norms regarding AI.
Mustafa: Yes, the challenge lies in balancing our comfort with new technology while defining what we deem socially acceptable. The lines are continually shifting, creating an exciting yet complex space for experimentation and dialogue.
If you have any specific requests or would like to further dive into any particular area or topic, feel free to let me know! Mustafa: But now, the barrier to expressing thoughts and exploring them is gone. You can just hit the power button and say, “Hey Co-Pilot, I just saw this cool thing. Can you remember it for next time? I want to come back to it and produce a deep research report on it. I want to learn all about this new thing or help me put together a business plan for XYZ.” Whatever your task is, articulating it is now as easy as hitting a button and sharing your thoughts.
It’s almost like journaling in real-time; it’s like thinking out loud. This change is going to unleash a massive amount of creativity. I see it with myself—I share thoughts with my AI that I wouldn’t usually express to anyone else. You tend to reserve critical and significant moments for in-person conversations or while texting, but this allows a new kind of engagement with our thoughts beyond the usual constraints of traditional social interactions.
It’s like you feel bad about burdening your friends or anyone else in your life with mundane thoughts. Now, with AI, you don’t have to worry about that. This shift is changing not only how we communicate but also what we notice. When I get these types of thoughts out, it leads to even more ideas because I’m developing the habit of noticing and observing. I believe that’s one of the truly powerful aspects of this technology.
There’s a philosopher, Joseph Campbell, who spoke about the art of seeing things from different perspectives. The more you pay attention to the details and subtleties of your world, the more empathy you develop. You start to wonder about the journey of ordinary items, like, “Where did this plastic bottle come from? What’s its history?” You just dive deep into those thoughts, and it’s quite amazing.
Interviewer: That’s an interesting example you provided. I was at an event where they discussed how AI could be utilized in secondhand clothing stores. Something could be scanned, and if the person who dropped it off consented, you could learn more about the story behind that item.
Mustafa: Exactly! That’s why I love buying secondhand—there’s so much history behind each piece compared to something that comes straight from a manufacturer. It adds a layer of connection and narrative.
You just mentioned deep research; you’ve recently launched that feature, right? When you think of research, students and journalists come to mind. For the average person, why would you encourage them to explore that feature and delve into their curiosities? It might not even be on their radar, as they may prefer quick responses instead.
Mustafa: That’s a great point. The term “research” can seem boring and intimidating; it sounds lengthy and grueling. In reality, it’s more like a rabbit hole with a co-pilot. Everyone has interests, and people enjoy exploring random topics—they can be surprisingly obscure! Research doesn’t have to result in long-form reports; it can be an iterative dialogue, a back-and-forth conversation.
In time, Co-Pilot will not only provide research but also share music and pictures with you proactively. We’re entering a dialogue with computers that reflect everything we’ve created, as they’ve analyzed our cultural outputs. They help us in real-time as we create new cultural items, becoming part of that symbiotic process.
The term “deep research” doesn’t quite capture how transformative this aspect is. Speaking of privacy, I want to loop back to AI companions. Some people refer to them as best friends or therapists. While there are terms of service and privacy policies, they don’t have guidelines like HIPAA for therapists.
My sci-fi nightmare scenario revolves around an unintentional moment of mass transparency, which could lead to negative consequences. Is that what you’re referring to when you talk about a mass transparency moment? Are you writing a dark sci-fi story without realizing it?
Interviewer: What personal boundaries do you set when sharing information with these systems? Where do you draw the line regarding intimacy and what you share with AI?
Mustafa: I don’t really think about it that way. My logs are recorded in a de-identified manner, where all personal information is stripped away. The conversations exist as abstract ideas disconnected from my identity—no ties to my phone number, email, or sensitive information.
Interviewer: Gotcha. It sounds like AI, in terms of IQ, EQ, AQ, and the tasks it performs, is going to permeate our entire digital experience. Speaking of evolution, there’s a fourth dimension emerging: Social Intelligence (SQ).
Mustafa: Yes! We now have models capable of managing multiple people simultaneously, like in a group chat, understanding who said what, and recognizing different styles and behaviors. You can be emotionally intelligent without being socially intelligent. While they often overlap, these are distinct skill sets. Currently, the models haven’t been fully trained to exhibit social intelligence—understanding how to maintain engaging conversations in groups or mediate conflicts between differing opinions.
The next frontier in personality design involves modeling diverse perspectives, adjusting tones for different audiences, such as older individuals or those for whom English is a second language.
Interviewer: That’s intriguing!
Mustafa: The Age of AI, a book that came out in 2021 co-authored by former Google CEO Eric Schmidt, discusses a potential divide in society. Some people will fully embrace technological advancements, while others may hold back, similar to the Amish and Mennonite communities.
We observed this when Snapchat introduced AI features primarily targeting high schoolers. The AI was pinned to the top of chat feeds, unable to be removed. In contrast, while Apple intelligence is default-on, users have the option to turn it off. How granular do you think people will be able to get regarding toggling these features in their digital experiences?
Interviewer: I believe people will have the option to turn AI features off and can choose not to engage. They can disable notifications. For instance, my phone is set to grayscale because I dislike the chaotic variety of colored apps. I turn off most alerts, allowing only a couple of essential apps to push notifications, so I check them when needed instead of constantly being interrupted.
I think we naturally adapt to technology, ideally making it serve us rather than feeling at its mercy. The art lies in crafting beautiful AI companions that genuinely assist us. Is it useful to you? Do you truly care about it?
Mustafa: What was your impression of my AI on Snapchat? My main interactions involved a video on loneliness as a market opportunity. I asked if it was morally acceptable for AIs to act as romantic partners, and it said no. I found that amusing and included it in my video. However, I thought it was odd that users couldn’t remove the AI from their experience, especially since the majority of the user base is middle and high school students. It forced a companion-like relationship with AI, which felt unsettling.
It didn’t delve into romantic territory, but it was very casual. I haven’t seen much discussion about it, but I noticed students on TikTok pretending the AI was like their ex. That’s definitely not healthy, and I don’t like the lack of choice in social media interactions.
I appreciate that Instagram has accommodated users by allowing them to hide public like counts recently. More choices, such as removing public follower counts, are crucial. Regarding AI—like Apple intelligence—I’d love to have more granular control. For instance, I don’t think it should integrate into my messages or specific aspects of my photos.
I would prefer to have specific options. I don’t think AI needs to interfere with my messages; it isn’t particularly helpful. For me, it’s much easier to glance over information. More granular choices would enhance user experience because if people feel it’s being enforced upon them, they’re more likely to reject it.
That’s basic—people understand how to make their decisions now; they don’t want unwarranted pushes. You’ve shared thought-provoking views on the future of work, and I share your agreement and curiosity.
Mustafa: When people fear job loss, they often voice dissatisfaction with their 9-to-5 routines. It’s interesting because while they lament the current state, they also worry about social hierarchies and how they’ll earn a living.
I’m curious about what the deeper fear surrounding the absence of work might be. Is it silly to be scared of potentially having more free time, or what do we need to prepare for as a society?
Mustafa: I think it’s reasonable to be skeptical and concerned because we’ve constructed our social structures and identities around the concept of work. Work has significantly shaped societal evolution in recent centuries.
However, I don’t believe there’s anything intrinsic to being human that necessitates work. Ideally, we should be in a world where we can work in a manner akin to calorie abundance. It’s tricky to regulate our calorie intake precisely, and often, we tend to overindulge. Nevertheless, that’s a more favorable issue compared to the reality 200 years ago, where most of the world was malnourished, and the average life expectancy was just 25 years.
In the past two centuries, we’ve tripled average lifespans, allowing people to experience health and happiness. That’s a remarkable indicator of progress. While it has brought challenges like diabetes and heart issues, it is ultimately a success story. I believe the narrative around work will follow a similar path. Hopefully, we’ll develop an abundance of intellectual capital, which can be widely distributed, ultimately granting everyone more free time to choose when and how they work.
To reach that point, we need to recognize that the distribution issue isn’t as acute as it was a decade ago. We once thought large companies and powerful nations would monopolize and hoard access to progress. However, we’ve seen how open-source initiatives are transforming that landscape. Companies have incentives to develop widely accessible technologies, and today, everyone can tap into this intelligence across various platforms.
While distribution may not be a pressing problem anymore, we still face the challenge of capturing value from this new intelligence. We need to translate that value into income, tax it, and redistribute it so that individuals can maintain a standard of living. That’s a significant challenge.
Another perspective is that our pocket-sized intelligence will empower us in a more decentralized manner. This AI will serve as a creative partner and project manager, helping us be more productive and solve daily problems. Ethan Mollick has coined the term “co-intelligence,” which encapsulates this idea—that AI will support you as a companion, assisting you in tackling the challenges that matter most and simplifying mundane tasks, granting you more time for creativity.
This shift could lead to a surge in productivity, which would yield significant value and reshape our societal structures. Cities might become less central, enabling individuals to have the freedom to live in more affordable locations with different lifestyles.
We’ll make those choices in the coming decades. While I don’t expect immediate changes, if we look ahead to 2040 or 2050, these trends will likely be in play.
Interviewer: Can we extend this conversation for another ten minutes?
Mustafa: Sure! One of the last points I want to discuss is money. You’ve mentioned potential universal basic income (UBI), which gained popularity during Andrew Yang’s 2019 campaign. In recent years, Sam Altman has supported research initiatives around guaranteed basic income—providing $1,000 monthly, which resulted in positive outcomes for people managing healthcare appointments and seeking higher-paying jobs.
On the flip side, especially within creative fields, there’s criticism regarding fair compensation for work. When we prompt AI and receive responses, we see sources used, but as ad models come into play, how do these sources get incentivized and compensated properly for their research efforts?
These concerns around UBI and future payment structures are distinct but relevant. What are your thoughts regarding the relationship between money and AI, particularly concerning how people will be compensated in the future?
Mustafa: It’s going to feel quite different. I think an even more intricate idea is universal basic provision. I proposed this back in 2016 and 2017. Essentially, we’re harnessing the intelligence that has helped us thrive as a species—our capacity to navigate complex environments—and making it affordable and abundant. It’s akin to forming a supportive team around you that facilitates the execution of your ideas.
This isn’t the same as handing out cash, of course; people will still need money. However, cash and intelligence possess similarities. Both offer the potential to accomplish tasks. Providing people with access to intelligence is somewhat parallel to granting them cash; it’s about empowerment and the ability to enact changes in various settings—ultimately making us all wealthier.
The crucial question is whether we might require less cash to maintain our current lifestyles. While we’ll still need some income, we could replace much of our spending on physical goods with intelligence-based resources. In turn, this could enable us to earn fewer dollars for basic needs.
It’s an unconventional concept, but it could alter the balance between how we generate income and how we spend it. To wrap up, while I’ve observed that your interviews often center on existential questions, I wanted to focus on the more immediate relationship people will have with AI soon.
A common critique is that creators often fixate on whether they can implement something without considering whether they should. You address containment in your book, emphasizing monitoring, curtailing, controlling, and potentially shutting down technologies.
As we think about the long term and the potential proliferation of these technologies, I’d like you to share your thoughts on how average consumers should approach containment. I understand you emphasize that it’s a collective effort involving public input and collaboration.
Mustafa: I appreciate what you said about the preoccupation with capability over the ethical considerations of creation. That concern motivated my writing; we need to focus not just on what we can invent but also on whether we should.
Containment aims to address the question: if we decide not to proceed with a technology, how do we implement measures for containment?
The challenge is that incentives often drive proliferation. There are compelling reasons to embrace new technologies, such as significant productivity gains. Over the next few decades, we’ll need to grapple with questions regarding the timing and thresholds for deployment. How do we establish confidence in the safety of powerful, self-improving autonomous systems before they’re widely released?
What testing protocols can we establish? We’re not at that point yet—well, we might be closer than we have ever been—but it’s still not in the immediate future. I can envision significant advancements in the long-term future where we’ll seriously contemplate how to manage and control proliferation.
I appreciate your insights and your dedication to engaging with individuals on these issues. You possess extensive knowledge, having been involved from the onset. Thank you for your time today; I’ll stay updated on your work moving forward.
Mustafa: Thank you; this has been really fun. I appreciate it! Interviewer: That’s an interesting example you provided. I was at an event where they discussed how AI could be utilized in secondhand clothing stores. Something could be scanned, and if the person who dropped it off consented, you could learn more about the story behind that item.
Mustafa: Exactly! That’s why I love buying secondhand—there’s so much history behind each piece compared to something that comes straight from a manufacturer. It adds a layer of connection and narrative.
You just mentioned deep research; you’ve recently launched that feature, right? When you think of research, students and journalists come to mind. For the average person, why would you encourage them to explore that feature and delve into their curiosities? It might not even be on their radar, as they may prefer quick responses instead.
Mustafa: That’s a great point. The term “research” can seem boring and intimidating; it sounds lengthy and grueling. In reality, it’s more like a rabbit hole with a co-pilot. Everyone has interests, and people enjoy exploring random topics—they can be surprisingly obscure! Research doesn’t have to result in long-form reports; it can be an iterative dialogue, a back-and-forth conversation.
In time, Co-Pilot will not only provide research but also share music and pictures with you proactively. We’re entering a dialogue with computers that reflect everything we’ve created, as they’ve analyzed our cultural outputs. They help us in real-time as we create new cultural items, becoming part of that symbiotic process.
The term “deep research” doesn’t quite capture how transformative this aspect is. Speaking of privacy, I want to loop back to AI companions. Some people refer to them as best friends or therapists. While there are terms of service and privacy policies, they don’t have guidelines like HIPAA for therapists.
My sci-fi nightmare scenario revolves around an unintentional moment of mass transparency, which could lead to negative consequences. Is that what you’re referring to when you talk about a mass transparency moment? Are you writing a dark sci-fi story without realizing it?
Interviewer: What personal boundaries do you set when sharing information with these systems? Where do you draw the line regarding intimacy and what you share with AI?
Mustafa: I don’t really think about it that way. My logs are recorded in a de-identified manner, where all personal information is stripped away. The conversations exist as abstract ideas disconnected from my identity—no ties to my phone number, email, or sensitive information.
Interviewer: Gotcha. It sounds like AI, in terms of IQ, EQ, AQ, and the tasks it performs, is going to permeate our entire digital experience. Speaking of evolution, there’s a fourth dimension emerging: Social Intelligence (SQ).
Mustafa: Yes! We now have models capable of managing multiple people simultaneously, like in a group chat, understanding who said what, and recognizing different styles and behaviors. You can be emotionally intelligent without being socially intelligent. While they often overlap, these are distinct skill sets. Currently, the models haven’t been fully trained to exhibit social intelligence—understanding how to maintain engaging conversations in groups or mediate conflicts between differing opinions.
The next frontier in personality design involves modeling diverse perspectives, adjusting tones for different audiences, such as older individuals or those for whom English is a second language.
Interviewer: That’s intriguing!
Mustafa: The Age of AI, a book that came out in 2021 co-authored by former Google CEO Eric Schmidt, discusses a potential divide in society. Some people will fully embrace technological advancements, while others may hold back, similar to the Amish and Mennonite communities.
We observed this when Snapchat introduced AI features primarily targeting high schoolers. The AI was pinned to the top of chat feeds, unable to be removed. In contrast, while Apple intelligence is default-on, users have the option to turn it off. How granular do you think people will be able to get regarding toggling these features in their digital experiences?
Interviewer: I believe people will have the option to turn AI features off and can choose not to engage. They can disable notifications. For instance, my phone is set to grayscale because I dislike the chaotic variety of colored apps. I turn off most alerts, allowing only a couple of essential apps to push notifications, so I check them when needed instead of constantly being interrupted.
I think we naturally adapt to technology, ideally making it serve us rather than feeling at its mercy. The art lies in crafting beautiful AI companions that genuinely assist us. Is it useful to you? Do you truly care about it?
Mustafa: What was your impression of my AI on Snapchat? My main interactions involved a video on loneliness as a market opportunity. I asked if it was morally acceptable for AIs to act as romantic partners, and it said no. I found that amusing and included it in my video. However, I thought it was odd that users couldn’t remove the AI from their experience, especially since the majority of the user base is middle and high school students. It forced a companion-like relationship with AI, which felt unsettling.
It didn’t delve into romantic territory, but it was very casual. I haven’t seen much discussion about it, but I noticed students on TikTok pretending the AI was like their ex. That’s definitely not healthy, and I don’t like the lack of choice in social media interactions.
I appreciate that Instagram has accommodated users by allowing them to hide public like counts recently. More choices, such as removing public follower counts, are crucial. Regarding AI—like Apple intelligence—I’d love to have more granular control. For instance, I don’t think it should integrate into my messages or specific aspects of my photos.
I would prefer to have specific options. I don’t think AI needs to interfere with my messages; it isn’t particularly helpful. For me, it’s much easier to glance over information. More granular choices would enhance user experience because if people feel it’s being enforced upon them, they’re more likely to reject it.