IMDA’s investigation found that the incident was within Singtel’s control, and it was not due to a cyber-attack. - Photo: ST
SINGAPORE: Telco Singtel has been fined S$1 million for a fixed voice service outage that affected 500,000 users for more than four hours on Oct 8, 2024.
Calls to some government agencies, healthcare organisations, banks and emergency hotlines 995 and 999 were disrupted.
“The potential impact on the safety and security of the public could have been very serious,” said sector regulator the Infocomm Media Development Authority (IMDA) on Thursday (Dec 11).
In determining the financial penalty, IMDA said it took into account the scale and impact of the disruption, and the time taken to restore the network.
In a statement on Dec 11, Singtel Singapore’s chief executive officer Ng Tian Chong said that the company accepts the financial penalty.
“We recognise the seriousness of the disruption to our fixed voice service last October and the concern and distress it caused our customers and members of the public. I would like to express my sincere apologies once again for the disruption and inconvenience experienced,” said Ng.
IMDA’s investigation found that Singtel hosted two separate virtualised firewalls – one for its fixed-line voice system and another for the monitoring system that manages home broadband routers and Pay TV set-top boxes – on the same hardware.
This meant that both virtualised firewalls shared the same memory resources.
On Oct 8, 2024, when the monitoring system had increased traffic, the shared memory of the hardware was overwhelmed. The monitoring system’s virtualised firewall did not have adequate filters to protect the hardware against high intensity traffic. This caused the virtualised firewall of the voice system to also malfunction and operate intermittently, said IMDA.
IMDA pointed out that voice traffic from the affected voice system should have been fully and seamlessly redirected to a separate voice system at an unaffected site through an automatic failover mechanism.
However, the failover did not happen seamlessly due to the intermittency of the affected voice system’s virtualised firewall. Calls were dropped intermittently as voice traffic alternated between the affected and unaffected voice systems.
The incident was resolved only after Singtel fully swung all voice traffic to the unaffected voice system, the authority added.
IMDA’s investigation also found that the incident was within Singtel’s control, and it was not due to a cyber-attack. These findings were supported by independent external consultants appointed to review the incident.
The authority added that the telco has since taken necessary remediation measures, including a separate hardware for its voice system and monitoring system, and an intervention mechanism to stop network traffic from alternating between its systems during failovers.
Ng said that Singtel also worked with its clients to strengthen their customer service hotline resiliency.
“Network reliability and resiliency is a top priority for us and we are committed to strengthening our systems, so that we can continue to provide reliable essential connectivity services to our customers,” said Ng.
IMDA said that it holds telcos which are key service providers, like Singtel, to a high level of service reliability.
“These service providers are required to plan, design and operate resilient networks, and put in place measures to ensure speedy recovery and minimal inconvenience to end-users in the event of a disruption,” said the authority.
The regulator has required other key service providers to conduct checks on their systems to avoid similar configuration weakness. It will validate the checks and remediation undertaken by the service providers.
IMDA added that it will not hesitate to take strong action under the Telecommunications Act, including imposing financial penalties, should any lapses be identified.
Under the Telecommunications Act, IMDA may impose a penalty of up to $1 million or up to 10 per cent of the annual turnover of errant service providers.
In Sep 2020, IMDA issued fines totalling $610,000 to M1 and Starhub for Internet service disruptions during the circuit breaker period of the Covid-19 pandemic.
Separately, M1 was imposed a financial penalty of $400,000 for broadband service disruptions in May 2020 that lasted a total of about 29 hours, while StarHub was fined $210,000 for intermittent disruptions in April 2020. - The Straits Times/ANN
