We’re well into the era of AI, with a variety of chatbots and digital assistants seeking to become permanent fixtures on our devices. Whether you like it or not, companies like Google are working hard to make sure AI becomes an integral part of the day-to-day experience. At the same time, the Malaysian government is pushing for widespread adoption of the tech as part of the country’s digital aspirations.
Of course, the ubiquitous presence of AI has shaped the way we engage with information online. Some users are growing increasingly dependent on the tech, sometimes to the detriment of themselves or their work. We’ve seen it before: instances of agents behaving in unintended ways, and the copious “hallucinations” confidently presented as fact. Despite these occasions proving that AI can be unreliable, people continue to place their trust in it.
An Amicable Assistant
It’s easy to brush this off as mere complacency, and that may be true to some extent. The tendency to accept and overly rely on automated systems is called automation bias. This happens due to a few reasons. For one, humans are inclined to take the path of least resistance. The less effort and energy spent, the better. This also applies to cognitive tasks. Then, there’s the view that these systems are superior in terms of analytical ability. You’re more likely to think the machine is better if you’re inexperienced.
With this in mind, the game isn’t exactly stacked in your favour. AI tools are designed in a way that feeds into our biases. Take Google’s AI Overviews, for instance. When you look something up in Google, it appears at the top of the results page, offering what looks like a handy dandy summary of all the information you need. It saves you the trouble of combing through the numerous links and poring over mountains of data. It’s quick. It’s simple. It’s convenient.
Never mind the little disclaimer suggesting you to double-check the details, or the links that are the supposed sources for said summaries. In most cases, people are looking things up casually, so they don’t feel the need to delve any deeper.
And then there’s the conversational chatbot format, which plays into another cognitive bias. Humans tend to equate linguistic capabilities with intelligence and reliability. ChatGPT and its ilk are coherent, so it’s easy to think of them as knowledgeable.
Additionally, these chatbots appeal to our social nature. They’re presented as amicable assistants, making them easy to anthropomorphise. Some interfaces also include some form of avatar designed to look cute or attractive. They adopt a friendly, likable tone, and as long as they don’t come off as too servile, people are inclined to trust them. Of course, with AI firms promoting the “memory” features that promise personalised experiences, it’s hard not to see how someone would believe that these bots are capable of caring. And there is some appeal in an all-knowing machine that “gets” you, even if it’s just smoke and mirrors.
The Cost Of Convenience
Now, one may wonder what exactly is the problem here. After all, isn’t the point of technological advancement to make our collective lives easier?
Except we’ve seen time and time again that the in-the-moment convenience AI offers isn’t without its consequences. Generative models have the tendency to produce inaccurate or outright incorrect information. It’s an unfortunate side effect of how these models are trained. Standard training rewards confident answers, even if they are false. On the flip side, honest uncertainty is punished. This means that the AI is incentivised to “lie”, because any answer is better than no answer. And again, these models aren’t really capable of reflecting on the morality of telling lies.
Although companies like OpenAI have acknowledged this flaw, solving it is easier said than done. Most fixes thus far address specific blunders, rather than the underlying issue. Arguably, hallucinations are integral to how these AI systems work. At base, they rely on predictive models that guess the most likely answer based on the training data. In short, they’re always making things up. It just so happens that in surface level interactions, the information tends to be accurate. It’s a whole other story once you get to the more specialised subjects.
And it’s not like companies are in a rush to fix this either. One way to deal with hallucinations is to train the models to admit when they do not have the answer. But saying “I don’t know” doesn’t quite fit the image of an omniscient assistant, and at the end of the day, these companies are pushing a product.
So what you get in the end are models that will spew nonsense at you and will only admit they are wrong when you push back hard enough. Obviously, if you’re not well-versed in a topic, you wouldn’t be inclined to question the bot. And the more complicated the task, the more likely the AI is to spit out garbage.
Embarrassing blunders aside, there is a subtler side effect of relying too much on AI. Studies over the past couple of years have looked into the influence of AI on its users. Essentially, AI use can inhibit skill formation, particularly if the person is counting on spoonfed answers. And these models tend to encourage passive reliance and cognitive offloading. Ultimately, heavy AI use can lead to a decline in critical thinking. This isn’t all too surprising, since people develop these skills through exploration and problem-solving. When using AI, all this is done by the bot for you.
The Bottom Line
So, what’s the takeaway here? Well, as AI continues to become more embedded into daily life, it’s increasingly important to recognise that these models are, at best, finicky tools. Despite appearances, they aren’t trustworthy companions one can take at face value 100% of the time.
Ultimately, when engaging with the tech, the best practice is to double check the bot’s output. One could argue that it’s better to avoid relying on the tech altogether, but that is somewhat of a tall order, given the continued push for its adoption. With the shift from generative AI that feeds you information to agentic AI that does things for you, it’s important to consider just how much trust you can place in the tech.
