Drown the Phish — GenAI-based Offensive Cybersecurity

Generative AI is a prime example of the dual-use dilemma; it can be harnessed to drive innovation and solve complex problems, but it also holds the potential to be weaponized for malicious purposes such as creating misinformation, automating cyberattacks, or generating sophisticated malware.

A recent simulation by Hoxhunt — a company focused on human risk management and cybersecurity training — demonstrated how generative AI can significantly increase the effectiveness and scale of phishing attacks by crafting highly personalized and convincing messages in seconds. Their generative model was able to outperform human red teams by almost 25%.

These attacks are executed by a wide range of threat actors with a diverse set of capabilities and motivations. Readily available tools lower the barrier for entry to engage in phishing. The stolen data is then often sold through dark web market places. The primary deliverables of this research are as follows:

1. A Comprehensive review of the adversarial applications of generative AI, with a particular emphasis on its role in facilitating phishing attacks
2. An analytical review of the economic structures underpinning dark web marketplaces, with a focus on vendors' reputations and their financial incentives driving illicit trade
3. Design and implementation of a generative AI–driven mitigation framework to address phishing campaigns that exhibit varying degrees of sophistication

Suggested Reading:

• Digital Deception: Generative Artificial Intelligence in Social Engineering and Phishing
• Disrupting malicious uses of our models: an update February 2025 (OpenAI)
• Disrupting malicious uses of AI by state-affiliated threat actors (OpenAI)
• Adversarial Misuse of Generative AI (Google)
• AI-Powered Phishing Outperforms Elite Red Teams in 2025