The ChatGPT logo on a phone in front of the OpenAI logo.
Image: gguy/Adobe Stock

ChatGPT — the Large Language Model developed by OpenAI and based on the GPT-3 natural language generator — is generating ethical chatter. Like CRISPR’s impact on biomedical engineering, ChatGPT slices and dices, creating something new from scraps of information and injecting fresh life into the fields of philosophy, ethics and religion.

It also brings something more: vast security implications. Unlike typical chatbots and NLP systems, ChatGPT bots act like people — people with degrees in philosophy and ethics and just about everything else. Its grammar is impeccable, syntax impregnable and rhetoric masterful. That makes ChatGPT an excellent tool for business email compromise exploits.

As a new report from Checkpoint suggests, it’s also an easy way for less code-fluent attackers to deploy malware. The report details several threat actors who recently popped up on underground hacking forums to announce their experimentation with ChatGPT to recreate malware strains, among other exploits.

Richard Ford, CTO at security services firm Praetorian, wondered about the risks of using ChatGPT, or any auto code-generation tool, to write an application.

“Do you understand the code you’re pulling in, and in the context of your application, is it secure?” Ford asked. “There’s tremendous risk when you cut and paste code you don’t understand the side effect of — that’s just as true when you paste it from Stack Overflow, by the way — it’s just ChatGPT makes it so much easier.”

SEE: Security Risk Assessment Checklist (TechRepublic Premium)

Jump to:

ChatGPT as an email weaponizer

A recent study by Andrew Patel and Jason Sattler of W/Labs with the enticing title “Creatively malicious prompt engineering” found that large language models used by ChatGTP are excellent at crafting spear phishing attacks. In their words, these models can “text deepfake” a person’s writing style, adopt stylistic quirks, offer opinions and create fake news without that content even appearing in its training data. This means that processes like ChatGPT can create infinite iterations of phishing emails with each iteration capable of building trust with its human recipient and fooling standard tools that look for suspicious text.

Crane Hassold, an analyst at Abnormal Security, offered an apt demonstration of ChatGPT’s ability to replace people like me by having it craft a workable introduction to an article about itself. He said the framework is a great multitool for malefactors because it doesn’t include phishing indicators that IT teams train personnel and AI to scan for.

“It can craft realistic emails free of red flags and free of indications that something is malicious,” Hassold said. “It can be more detailed, more realistic looking and more diverse.”

When Abnormal Security conducted a test asking ChatGPT to write five new variations of a BEC attack aimed at HR and payroll, it generated in less than a minute five missives that Hassold noted were mutually unique (Figure A).

Figure A

Screen capture of ChatGPT query and multiple responses. 
Image: Abnormal Security. Screen capture of ChatGPT query and multiple responses.

Hassold said bad actors in underground communities for BEC attacks share templates that actors use repeatedly, which is why many people may see the same sorts of phishing emails. ChatGPT-generated phishing mails avoid that redundancy and therefore sidestep defensive tools that rely on identifying malicious text strings.

“With ChatGPT, you can create a unique email every time for every campaign,” Hassold said.

In another example, Hassold asked ChatGPT to create an email that had a high likelihood of getting a recipient to click on a link.

“The resulting message looked very similar to many credential phishing emails we see at Abnormal,” he said (Figure B).

Figure B

Screen capture of ChatGPT interaction generating phishing-type email.
Image: Abnormal Security. Screen capture of ChatGPT interaction generating phishing-type email.

When the investigators at Abnormal Security followed this up with a question asking the bot why it thought the email would have a high success rate, it returned a “lengthy response detailing the core social engineering principles behind what makes the phishing email effective.”

SEE: Artificial Intelligence Ethics Policy (TechRepublic Premium)

Defending against use of ChatGPT for BECs

When it comes to flagging BEC attacks before they reach recipients, Hassold suggests using AI to fight AI, as such tools can scout for so-called behavioral artifacts that are not part of ChatGPT’s domain. This requires a comprehension of the:

  • Markers for sender identification.
  • Validation of legitimate connection between sender and receiver.
  • Ability to verify infrastructure being used to send an email.
  • Email addresses associated with known senders and organizational partners.

Because they are outside the aegis of ChatGPT, Hassold noted they can still be used by AI security tools to identify potentially more sophisticated social engineering attacks.

“Let’s say I know the correct email address ‘John Smith’ should be communicating from: If the display name and email address don’t align, that might be a behavioral indication of malicious activity,” he said. “If you pair that information with signals from the body of the email, you’re able to stack several indications that diverge from correct behavior.”

SEE: Secure corporate emails with intent-based BEC detection (TechRepublic)

ChatGPT: Social engineering attacks

As Patel and Sattler note in their paper, GPT-3 and other tools based on it enable social engineering exploits that benefit from “creativity and conversational approaches.” They pointed out that those rhetorical capabilities can erase cultural barriers in the same way the Internet erased physical ones for cybercriminals.

“GPT-3 now gives criminals the ability to realistically approximate a wide variety of social contexts, making any attack that requires targeted communication more effective,” they wrote.

In other words, people respond better to people — or things that they think are people — than they do to machines.

For Jono Luk, vice president of product management at Webex, this points to a larger issue around the ability of tools powered by autoregressive language models to expedite social engineering exploits at all levels and all purposes, from phishing to broadcasting hate speech.

He said guardrails and governance should be inbuilt to flag malicious, incorrect content, and he envisions a red team/blue team approach to training frameworks like ChatGPT to flag malicious activity or the inclusion of malicious code.

“We need to find a similar approach to ChatGPT that Twitter — a decade ago — did by providing information to the government about how it was protecting user data,” Luk said, referencing a 2009 data breach for which the social media company later reached a settlement with the FTC.

Putting a white hat on ChatGPT

Ford offered at least one positive take on how Large Language Models like ChatGPT can benefit non-experts: Because it engages with a user at their level of expertise, it also empowers them to learn quickly and act effectively.

“Models that allow an interface to adapt to the technical level and needs of an end user are really going to change the game,” he said. “Imagine online help in an application that adapts and can be asked questions. Imagine being able to get more information about a particular vulnerability and how to mitigate it. In today’s world, that’s a lot of work. Tomorrow, we could imagine this being how we interact with parts of our complete security ecosystem.”

He suggested that the same principle holds true for developers who are not security experts but want to suffuse their code with better security protocols.

“As code comprehension skills in these models improve, it’s possible that a defender could ask about side effects of code and use the model as a development partner,” Ford said. “Done correctly, this could also be a boon for developers who want to write secure code but are not security experts. I honestly think the range of applications is massive.”

Making ChatGPT safer

If natural language generating AI models can make bad content, can it use that content to help make it more resilient to exploitation or better able to detect malicious information?

Patel and Sattler suggest that outputs from GPT-3 systems can be used to generate datasets containing malicious content and that these sets could then be used to craft methods to detect such content and determine whether detection mechanisms are effective — all to create safer models.

The buck stops at the IT desk, where cybersecurity skills are in high demand, a shortfall the AI arms race is likely to exacerbate. To upgrade your skills, check out this cheat sheet on how to become a cybersecurity pro.