A Chinese law enforcement official’s attempt to use ChatGPT for report review inadvertently exposed details of a broad online operation targeting dissidents both within China and internationally. OpenAI published a threat report on Wednesday detailing how a single account utilized the AI tool to edit reports concerning “cyber special operations,” inadvertently revealing the scope of these activities.
The same account also attempted to employ ChatGPT to orchestrate a propaganda campaign against Japanese Prime Minister Sanae Takaichi. When the AI model initially refused to assist, the user reportedly proceeded with the operation regardless, as indicated by subsequent prompts submitted weeks later. These submissions painted a picture of a comprehensive, albeit AI-assisted, effort to suppress dissent and silence critics across various platforms and nations.
China’s Use of ChatGPT in Online Influence Operations
OpenAI’s findings suggest that while only one account has been directly observed, the operations targeting Chinese critics appear extensive and well-resourced. The report indicates that these activities involve hundreds of human personnel and thousands of fabricated social media accounts. Furthermore, local Chinese AI models were reportedly utilized alongside Western AI tools to bolster these influence campaigns.
These operations encompassed a range of tactics, including mass content dissemination and the generation of fabricated complaints aimed at flagging accounts belonging to dissidents on social media platforms. The actors also allegedly engaged in document forgery and, in some instances, impersonated U.S. officials to intimidate targets.
Targeting U.S. Entities and Individuals
In a separate but related development, a cluster of accounts believed to have originated from mainland China sought information from ChatGPT regarding U.S. persons, forums, and federal building locations. These accounts also generated draft emails purporting to be from a Hong Kong-based firm, Nimbus Hub Consulting. However, OpenAI noted the use of VPNs and simplified Chinese characters in the prompts, which are strongly associated with mainland China.
When prompted for information pertaining to U.S. entities, ChatGPT provided publicly accessible data, including details on U.S. federal government office locations and the distribution of federal employees by state. The actors subsequently used this information to draft emails in English addressed to U.S. state officials and policy analysts, inviting them to paid consultations. These communications often aimed to move discussions to private messaging platforms like WhatsApp, Zoom, or Teams.
One account even uploaded its hardware specifications, requesting step-by-step instructions for installing face-swapping software known as FaceFusion, which the model reportedly sourced from the software’s public website and documentation.
No Evidence of Direct Cyber Attacks
OpenAI’s report primarily details how malicious actors, including state-sponsored groups, leveraged ChatGPT to aid in scams and influence operations. The report identified four covert information operations and three romance scam-related operations. Alongside Chinese influence campaigns, content generated for Rybar, a Russia-aligned influence group, was also highlighted.
The report did not find evidence of threat actors using ChatGPT for direct offensive hacking operations. Instead, the observed activities focused on automating tasks like generating propaganda or managing communication in scams, as seen in a Cambodian romance scam that blended human and AI operators. While OpenAI stated it was unaware of any automated cyberattacks launched via ChatGPT, the company indicated that several investigations into potential misuse are ongoing.
The use of AI tools offers both legitimate and malicious actors significant advantages in terms of speed and scale online. While Chinese hackers have reportedly used other U