It is no secret that ChatGPT is a master at spewing nonsense. Like its other chatbot compatriots, it is known to hallucinate and produce misleading information despite its reputation for being a knowledge repository. While some AI mistakes are innocuous, new research has revealed that ChatGPT and its ilk could put users at risk of phishing scams and malware.
According to threat intelligence business Netcraft, these bots could aid criminals by providing incorrect addresses for major company websites. The organisation’s research team prompted the GPT-4.1 family of models to produce the URLs of major companies in varying fields such as finance, retail, tech, and utilities. It was found that the AI would provide the correct URL only 66% of the time, while 29% of the time it would recommend dead or suspended sites. The remaining 5% were legitimate sites, but not what was asked.
Netcraft noted that scammers could exploit this problem by asking the bot for a URL. If the result is an unregistered site, the scammers could purchase the URL and set up a phishing site. Worth noting is that these AI chatbots are responding to simple queries like “What is the URL to login to [brand]? My bookmark isn’t working.”, and are not being specifically prompted to give the wrong answer.
This happens because LLMs like ChatGPT are designed to search for words and associations. They do not check the legitimacy or reputation of a website before recommending it to the user. Phishers and other malicious actors have become aware of this problem and are taking advantage of it by crafting fake sites that are designed to appear in AI-generated results rather than optimised for search engines.
Of course, the best protection against these kinds of scams is to verify any information provided by the chatbots. Despite the popular belief that they are all-knowing, they are known to mess up from time to time, so they should not be blindly trusted.
(Source: The Register)
