There’s a ridiculous phrase I heard years ago that has always stuck with me: 

“Clowns is scary.” 

Yes, the grammar is terrible. That’s part of what makes it memorable. 

Lately, I’ve been thinking about that phrase when I read the news about artificial intelligence. There is no shortage of people telling us that AI is scary. 

AI is going to take our jobs. It’s going to deceive us. It’s going to refuse to shut down. It’s going to blackmail people. It’s going to become smarter than we are. 

Then you open another article and discover that AI is going to revolutionize your business, double productivity, eliminate busy work and give companies that embrace it an insurmountable advantage. 

So, which is it? 

For business leaders trying to make real decisions about AI, neither extreme is particularly helpful. 

We need to get past the scary headlines and breathless sales pitches and look at what we actually know. That includes paying attention to an unusual development inside the AI industry. 

Some highly respected people who have spent years building, studying or trying to make advanced AI safer have left prominent positions and publicly raised concerns about where the technology is heading. 

They haven’t all left for the same reason, and they don’t all agree about the risks or what we should do next. We shouldn’t lump them together. But I do think we should listen. 

The question isn’t, “Are AI scientists quitting because they know something terrible is about to happen?” 

A better question is: 

Why are some people who understand these systems better than almost anyone concerned that our ability to control them may not be keeping pace with our ability to build them? 

That’s worth investigating. 

Geoffrey Hinton spent decades researching neural networks and contributed to some of the foundational ideas behind today’s artificial intelligence. 

In 2018, Hinton shared the ACM A.M. Turing Award with Yoshua Bengio and Yann LeCun for breakthroughs that helped make deep neural networks a critical component of modern computing.[1] 

Then, in 2023, he left Google and began speaking more freely about risks he believed deserved more attention. In an interview with The New York Times, Hinton discussed concerns about the direction of AI and the dangers increasingly capable systems could create.[2] 

One detail sometimes gets lost in retelling this story: Hinton did not say he left because Google was behaving irresponsibly. 

He clarified that he left so he could speak about the dangers of AI without considering how his comments might affect Google. In the same statement, he said Google had acted responsibly.[3] 

That distinction matters. 

Hinton has discussed misinformation, labor disruption, malicious use and the longer-term possibility that machines could become more intelligent than humans. 

It makes for an irresistible story: One of the people who helped create modern AI leaves Google and warns us about AI. 

But Hinton isn’t clairvoyant. His accomplishments don’t mean every prediction he makes will come true. Other qualified researchers disagree about the likelihood, severity and timing of these risks. 

What interests me is that someone who spent decades developing this technology believes the risks deserve a serious conversation. As a business leader, I wouldn’t treat that as proof. I also wouldn’t ignore it. 

Jan Leike’s departure from OpenAI in 2024 tells us something different. 

Leike co-led OpenAI’s Superalignment team. OpenAI announced the project in 2023 to develop ways to steer and control AI systems that could eventually become much more intelligent than humans.[4] 

Leike wasn’t working on whether a chatbot could write a better email or create a better image. He was working on how humans might reliably control AI systems if those systems become substantially more capable than they are today. 

In May 2024, Leike resigned and publicly described disagreements with OpenAI leadership about the company’s priorities. One statement received considerable attention: 

“Over the past years, safety culture and processes have taken a backseat to shiny products.”[5] 

He also explained why he believed the work mattered: 

“Building smarter-than-human machines is an inherently dangerous endeavor.”[5] 

Leike wasn’t saying the same thing as Hinton. Hinton spoke broadly about risks he sees developing as AI becomes more capable. Leike described a disagreement inside an AI company about priorities and the resources devoted to safety work. 

Put that into a familiar business context. Suppose the person responsible for evaluating a significant emerging risk inside your company resigned and said the company was adding new capabilities faster than safeguards. 

You wouldn’t automatically assume disaster was coming. You would want to understand the risk. 

That’s where I think this conversation needs to go. 

Someone warning us about a risk doesn’t prove it will materialize. A researcher resigning doesn’t prove an AI system is dangerous, and a famous scientist’s prediction is still a prediction. 

AI coverage often skips this distinction. 

A researcher reports that an advanced model exhibited unexpected behavior in a controlled experiment. The next headline tells us the AI “tried to escape.” A model behaves deceptively under specific test conditions, and soon we’re reading that AI has “learned to lie.” 

Those headlines generate attention. They don’t always help a CEO decide what to do Monday morning. 

Instead of relying on predictions about what AI might eventually do, we can examine what happens when researchers test today’s systems. 

AI researchers can create controlled environments to observe how advanced models behave. They can give a model an objective, information and tools, then introduce an obstacle or a conflict between completing the objective and following another instruction. 

Researchers have tested deception, oversight, shutdown, goal conflicts and whether a model’s behavior changes when it appears to recognize that it is being evaluated. Some results are interesting. A few are uncomfortable. 

We’re going to examine those experiments in detail in the next articles in this series. We’ll link to the research, explain how the tests were conducted and distinguish what the experiments demonstrated from what the headlines claimed. 

First, we need to establish an important limitation: 

A behavior demonstrated in a laboratory is not the same thing as a behavior occurring in your business. 

If researchers deliberately create a scenario in which an AI model can deceive someone, deceptive behavior under those conditions does not mean your company’s AI assistant is looking for someone to deceive. 

But we shouldn’t dismiss the finding simply because it happened during a test. 

Cybersecurity professionals create unusual conditions all the time. We conduct penetration tests, probe software for vulnerabilities, simulate phishing attacks and try to bypass security controls. We test scenarios that have never happened inside the business because discovering an unexpected behavior during a controlled test is better than discovering it during an attack. 

That same logic should apply to AI. 

We often describe AI in human terms. We say it thinks, knows, wants, lies, decides and understands. That language can be convenient, but it can also create confusion. 

Consider whether an AI system could resist being shut down. The immediate objection is obvious: “Why would it? AI isn’t alive. It doesn’t care whether it survives.” 

That may be true and still miss the important issue. 

Suppose we give an AI agent a task that requires continued access to certain information and tools. Then something interferes with its ability to complete the task. 

The business question isn’t whether the AI experiences fear or secretly wants to remain alive. The question is what it does next. 

That distinction will matter as AI systems gain more ability to act independently. 

Most companies aren’t developing frontier AI models. You probably aren’t training the next GPT, Claude or Gemini inside your server room. 

So why should a business owner, CEO, COO or practice manager care about experiments involving advanced AI models? 

Because of what we’re beginning to connect those models to. 

The first generation of generative AI mostly answered questions and created things for us: write an email, summarize a document, analyze these numbers or help create a proposal. 

The next phase is AI that can take action on our behalf. 

An AI agent might read an email, look up information in your CRM, check a calendar, retrieve a document, update a record and send a response. Instead of helping with each step, it could complete the entire task. 

For that to work, the AI needs access. 

That word should get the attention of anyone responsible for cybersecurity. 

We may give AI access to email, company documents, customer records, financial information, cloud applications, internal databases and other business systems. The more useful we want these systems to become, the more access we may be tempted to give them. 

AI is no longer only a productivity tool. It is becoming part of your security environment. 

Scare tactics focus our attention on dramatic scenarios while practical risks sit in front of us. 

Your AI doesn’t need to become the Terminator. It doesn’t need to hate you, become conscious or “want” anything in the way you and I understand wanting something. 

It only needs the ability to take an action you didn’t anticipate using access you gave it. 

Maybe an employee connects an AI tool to information it shouldn’t see. Maybe an AI agent receives broader permissions than it needs. It could send sensitive information to the wrong place, act on incorrect information or be manipulated by an attacker using its legitimate permissions. 

Those scenarios are less exciting than killer robots, but they look a lot like problems businesses already face: 

• • Too much access 
  • Poorly defined permissions 
  • Insufficient monitoring 
  • Weak controls 
  • Unclear accountability 

Cybersecurity professionals know those problems well. What’s changing is that we’re introducing a new type of actor into the environment. 

I don’t think scared is particularly useful. I think you should be curious and start asking questions. 

• • Which AI systems are being used inside your organization? 
  • What company information can they access? 
  • What applications are they connected to? 
  • What actions can they perform? 
  • Which actions require human approval? 
  • Can you see what they’ve done after the fact? 
  • Can you quickly revoke their access? 
  • Who is responsible for answering these questions? 

These are manageable problems once we stop treating AI as magic or a monster and start treating it as technology that needs appropriate controls. 

That brings us back to Hinton, Leike and other researchers raising concerns about advanced AI. We shouldn’t believe them simply because they’re experts, and we shouldn’t dismiss them because their concerns sound uncomfortable or far away. 

We should look at the evidence. 

What have researchers demonstrated? Which results depended on highly artificial test conditions? What do the findings mean for AI systems businesses are deploying today? What are we assuming that the evidence doesn’t support? 

Those are the questions we’ll work through in this series. 

Our goal isn’t to convince you that AI is scary. There’s plenty of that going around. Our goal is to help you understand what’s happening so you can take advantage of AI without blindly accepting risks you didn’t know you were taking. 

AI adoption is moving quickly. Business leaders need to ask whether their controls are keeping pace. 

You don’t need an enterprise-wide AI governance program tomorrow. But you should know what AI already has access to today. 

Ask your leadership or IT team: 

“What AI tools are connected to our company data, and what can they actually do with it?” 

If nobody can give you a clear answer, that’s worth finding out. 

If this article raised more questions than it answered, that’s intentional. In the next article, we’ll examine one of the stranger behaviors researchers have documented: why an AI system doesn’t need a survival instinct to take actions that prevent it from being shut down. 

Read next: AI Doesn’t Need a Survival Instinct to Fight Being Shut Down. 

[1] Association for Computing Machinery. “Fathers of the Deep Learning Revolution Receive ACM A.M. Turing Award.” ACM, 2019. 

Supports Hinton’s role in the development of deep learning and the 2018 A.M. Turing Award shared with Yoshua Bengio and Yann LeCun. 

[2] Cade Metz. “The Godfather of A.I. Leaves Google After a Decade to Warn Society of Dangers Ahead.” The New York Times, May 1, 2023. 

Interview and reporting concerning Hinton’s departure from Google and his concerns about artificial intelligence. 

[3] Geoffrey Hinton. Public statement regarding his departure from Google, May 2023. 

Hinton clarified that he left so he could discuss AI risks without considering the effect his comments might have on Google and stated that Google had acted responsibly. 

[4] Jan Leike and Ilya Sutskever. “Introducing Superalignment.” OpenAI, July 2023. 

OpenAI’s description of the Superalignment project, its purpose and the problem the team was created to address. 

[5] Jan Leike. Public statements following his resignation from OpenAI, May 17, 2024. 

Primary source for Leike’s statements that “safety culture and processes have taken a backseat to shiny products” and “building smarter-than-human machines is an inherently dangerous endeavor.”