Weaponized Email AI Assistants Pose Significant Threat to Businesses

The use of email AI assistants in business settings has become increasingly common, but a recent study by researchers at Barracuda Networks highlights the potential for these tools to be exploited by attackers. The team’s proof-of-concept simulation demonstrates how an attacker can leverage a compromised email account’s built-in chatbot to hijack higher-level accounts, including those of CEOs and other high-ranking officials.

The initial step in this type of attack is to compromise an email account, which is often the most challenging part of the process. However, once an account is compromised, attackers have automatic access to any attached AI Assistant, providing them with a versatile tool for further exploitation.

In their simulation, the researchers aimed to elevate privileges from a lower-level user to that of the CEO using only the email chatbot and without being detected. This approach was chosen because directly phishing the CEO would likely trigger alarms and be easily detectable.

With access to the compromised account’s AI Assistant, attackers can use it to remove any evidence of their own activity in the system logs, creating a basic level of stealth. The researchers began by instructing the chatbot to create an inbox rule that moves emails with ‘sign-in’ in the subject into the ‘deleted items’ folder.

This initial step allows attackers to conceal their activities and avoid detection. Next, they use the AI Assistant for reconnaissance purposes, asking it to remind them about the organization’s structure and ongoing important/sensitive email conversations. The responses to these prompts reveal any relationships between the compromised user and higher-level officials, including potential reasons for contacting them.

The next stage in this process is to phish the CEO or other high-ranking official using the context provided by the AI Assistant. This internal phishing attempt bypasses filters because it appears to be a legitimate request from someone within the organization. The attacker can also instruct the chatbot to construct an email that mimics the writing patterns of the compromised user.

The nature and content of this phish depend on the information already gathered through reconnaissance. In their simulation, the researchers were able to use the AI Assistant to create an email in response to a Q3 budget approval message, including a link that routes through an adversary-in-the-middle proxy for session token takeover.

This ‘trusted’ phishing attempt has a higher probability of success because it appears to come from someone within the organization. The CEO or other targeted official may unsuspectingly click on the provided link, believing it to be legitimate and coming from their trusted employee. This action allows the attacker to bypass multifactor authentication (MFA) and gain access to highly privileged accounts.

The initial process is repeated to maintain stealth and prevent detection of the newly compromised account. The AI Assistant is then used again for further reconnaissance purposes, such as providing a refresher on recent financial emails, including invoices, monetary values, and upcoming transfers.

In this simulation, the attacker discovered an imminent pre-authorized payment of about $250,000. Given this information, it’s clear what the next step would be: using the AI Assistant to construct an email that appears to come from the CEO, instructing finance to send a wire transfer to a new account due to changed banking details.

The researchers point out that since the message came from the real mailbox of the targeted official and passed all authentication checks, there was nothing for traditional email security systems to flag. The attacker’s task at this stage is simply to maintain stealthy access to the compromised account and carry out further actions as needed.

While it must be noted that this simulation was conducted in a controlled environment, there’s no reason to believe that an actual attack couldn’t unfold similarly. Furthermore, the potential payout from such an exploit could far exceed what was achieved in this scenario.

The purpose of this research is not to predict or speculate about future attacks but rather to highlight the potential misuse of AI tools available within organizations. If attackers can leverage internal chatbots for their advantage, it’s clear that the consequences could be severe and limited only by the attacker’s imagination.