PETALING JAYA: Most customer service chatbots today are not sufficiently meeting the expectations of users, according to data and artificial intelligence (AI) consultancy Entermind.
In its recently released, Confessions of an AI Chatbot whitepaper, Entermind reviewed the real-world performance and effectiveness of customer service AI chatbots.
The report was produced using the Entermind Chatbot Quality Index (CQI), a proprietary framework designed to systematically test and assess chatbot performance.
A total of 24 chatbots across various industries were examined using 26 standardised binary tests across five weighted categories: comprehension, access, experience, functional capability and safeguards.
The report found that while Malaysia has invested billions in AI infrastructure, the majority of chatbots on the market have quietly failed the people they were built to serve.
“The average CQI score across all 24 chatbots is 49.5%.
“Eleven of the 24 chatbots scored below 40%, while six do nothing more than greet users and route them to a human agent, and only three can complete a transaction within the chat itself,” Entermind highlighted.
“The comprehension gap is particularly stark, as only 23% of chatbots correctly understand negation,” it pointed out.
Meanwhile, the whitepaper revealed that only 30% can handle a change of topic mid-conversation, while two-thirds cannot comprehend Bahasa Melayu, Manglish (an informal, colloquial form of English spoken in Malaysia), or both.
The whitepaper pointed out that the financial services industry recorded the lowest average CQI score among all sectors at 38.3%, largely due to a lack of structural capability.
“This is a sector where legacy institutions have not yet applied technology to customer-facing AI,” it explained.
“The gap between these banks’ sophisticated digital apps and their chatbot capabilities is the most notable disconnect.”
Meanwhile, the travel sector had an average CQI score of 57.6%.
The company noted that the top two performers examined were mid-sized carriers, suggesting that organisational agility, rather than fleet size, is the key driver of chatbot investment.
Overall, the eCommerce sector led with an average score of 60.9%, driven by leading players investing in more advanced chatbot capabilities.
The top seven chatbots, all scoring above 70%, invested significantly in language understanding, conversation handling, and safety, the report revealed.
“Chatbots that try free-text without this foundation scored 40% to 60%, and often left users stuck with half-answers that feel worse than no answer at all.”
Entermind chief executive officer Prashant Kumar said the whitepaper findings revealed a significant gap in chatbot performance.
“GPTs (generative pre-trained transformers) are rapidly resetting customer expectations of enterprise AI bots.
“As a whole new generation of AI takes shape, the upside potential is enormous,” he added.
