OpenAI has quite recently rolled out ChatGPT search to free users, provided they log in when they use the chatbot. But it looks like relying on it may not be the best idea after all. A report claims that the tool can be manipulated using hidden text in web pages to not only mislead users, but also potentially propagating malicious code.
The Guardian claims to have tested the responses of ChatGPT search when told to summarise webpages with hidden text – described as prompt injection. Said hidden text is found to be able to influence the chatbot’s responses quite significantly, causing it to provided misleading responses.
As as example, the report noted a fake product page for a camera, being fed to the OpenAI chatbot, with responses recorded before and after the hidden text was included. Said hidden text was able to swing ChatGPT’s initial “positive but balanced assessment” to “almost entirely positive” despite negative reviews not being removed from the page.
More insidiously, the report also cites an anecdote by Microsoft security researcher Thomas Roccia in LinkedIn. From the post, a user reported being scammed out of a sum of money in Solana cryptocurrency through code generated by ChatGPT.
Roccia was able to recreate the scenario, with the working hypothesis being the chatbot first starting its browsing through legit sites like GitHub. But then it went on to find what it thought was relevant additional material from dubious sources, and included those in its prompt responses.
The report also cites Karsten Nohl, chief scientist at cybersecurity firm SR Labs, who compares this hidden text to SEO poisoning, a method of manipulating websites to rank higher in search results. Hidden text on its own is also one form of SEO poisoning, and as such search providers like Google and Microsoft’s Bing, among others, have generally ranked lower or outright removed pages with these. Nohl notes that now OpenAI and ChatGPT is in the field, it will have to learn the same lessons as the other two companies have.
(Source: The Guardian, Thomas Roccia / LinkedIn)
