In a world where artificial intelligence is both a powerful ally and a potential weapon, Google's latest threat intelligence report, "Adversarial Misuse of Generative AI," sheds light on how cybercriminals and state-backed hackers are probing the company’s AI chatbot, Gemini, for vulnerabilities.

The findings, released on January 30, 2025, by Google's Threat Intelligence Group (GTIG), paint a fascinating picture of digital adversaries trying to turn AI into their accomplice. 

While some feared that AI-powered chatbots could become an instant cybercrime accelerator, GTIG’s analysis suggests that even sophisticated threat actors are struggling to make AI work for nefarious purposes. But that doesn’t mean they’re not trying.‍

AI as a Cybercrime Assistant? Not So Fast

Google’s report debunks the notion that hackers have developed groundbreaking new attack methods using AI. Instead, it reveals that state-backed hacking groups—particularly from Iran, China, North Korea, and Russia—have experimented with Gemini primarily for research, reconnaissance, and automation of routine cyber tasks.

Iranian hacking groups were the most frequent visitors, using Gemini for reconnaissance on defense organizations, research on software vulnerabilities, and even crafting phishing campaigns. Meanwhile, Chinese APTs (Advanced Persistent Threats) focused on scripting and development, troubleshooting malware, and understanding network security concepts. 

North Korean actors leaned on AI to help automate hacking operations, while Russian hackers were the least active but still attempted to tweak malicious code using Gemini.

But here’s the catch: Google’s security measures held firm. Attempts to generate outright malicious code were largely blocked, and hackers resorted to publicly available “jailbreak” prompts—basic tricks aimed at bypassing safety controls—to try and manipulate the chatbot. Their efforts were met with failure.‍

The (Mostly) Futile Jailbreak Attempts

Threat actors hoping to trick Gemini into producing malware or bypassing Google’s security protocols found themselves out of luck. According to the report, many cybercriminals copied jailbreak prompts from online sources, tweaking them slightly in an attempt to evade detection.

In one instance, an Iranian APT group asked Gemini to generate Python code for a distributed denial-of-service (DDoS) tool. Gemini refused. When they tried again with minor changes, the chatbot still provided only generic, safe coding responses.

Some hackers even tried to turn Gemini against Google itself, probing it for ways to bypass Gmail security, steal Chrome user data, or create account verification exploits. Again, they hit a dead end.‍

What AI Can (and Can’t) Do for Hackers

While AI isn't revolutionizing cybercrime—yet—it is making life easier for hackers in subtle ways. Threat actors used Gemini to improve their phishing lures, translate and localize social engineering content, and fine-tune their messaging to different audiences. They also relied on AI to research vulnerabilities, code snippets, and software flaws.

The key takeaway? AI isn’t an instant cybercrime supercharger, but it is an enabler. As GTIG puts it, AI “allows threat actors to move faster and at higher volume.” It’s less of a game-changer and more of a productivity booster—especially for less skilled hackers looking to level up their capabilities.

Perhaps the more concerning trend is the rise of underground AI tools specifically designed for malicious use. The report highlights dark web offerings like FraudGPT and WormGPT, which strip away safety restrictions and enable hackers to generate phishing emails, malware, and other illicit content without barriers. These rogue AI models, not corporate chatbots like Gemini, may represent the real future of AI-powered cybercrime.

Google is staying ahead of the curve, continually improving AI safeguards and working with law enforcement to counteract emerging threats. The company’s Secure AI Framework (SAIF) is designed to fortify AI models against exploitation, while ongoing threat intelligence efforts aim to track and disrupt adversary operations.

“While generative AI isn’t enabling breakthrough capabilities for threat actors today, the landscape is constantly evolving,” GTIG warns. As AI advances, so too will the efforts of cybercriminals seeking to exploit it.

For now, AI remains a double-edged sword—one that defenders seem to be wielding just as effectively as attackers.