SINGAPORE, April 21 — Following a recent technical glitch which sparked hours-long delays and snaking queues at Singapore’s checkpoints, the Immigration and Checkpoints Authority (ICA) is reviewing its processes when it comes to upgrades to its immigration clearance system.
Around 85,000 travellers were affected — including 21 passengers who missed their flights — as a result of the close to four-and-a half-hour issue on March 31, which started at around 10.40am.
Minister of State for Home Affairs and National Development Assoc Prof Muhammad Faishal Ibrahim said this in Parliament today (April 21) as he gave details on what had caused the glitch, in response to several parliamentary questions from Members of Parliament.
What happened
On the morning of the incident, the ICA was conducting a pre-scheduled trial that was needed for an upgrade of a system, which facilitates automated immigration clearance using travellers’ biometrics.
Unfortunately, the trial caused a severe system overload which impacted the automated clearance lanes at all the departure halls in Changi Airport and some automated lanes at Woodlands and Tuas Checkpoints.
The rest of the automated lanes, including those at the sea checkpoints, remained operational, said Assoc Prof Faishal.
He added that during the disruption, about 55,000 travellers passed through the two land checkpoints, while about 30,000 passengers on 113 departure flights and 111 arrival flights, cleared immigration at Changi Airport. He did not give details on how many travellers would have cleared immigration under normal circumstances.
Prior to the incident, he said ICA had been upgrading its systems “progressively and cautiously”, with 10 out of 12 systems enhanced.
For the system that was being upgraded on March 31 — called the multi-modal biometrics system (MMBS) — the upgrade involved replicating the large biometric database to an upgraded system.
“This has to be done continuously over a few days, and the MMBS has to remain operational during this period,” he said.
Assoc Prof Faishal added that any system upgrade typically undergoes a trial test first at different times of the day, before proceeding with the actual upgrade.
Prior to the trail on March 31, a trial was successfully conducted on March 15 from 1.30am to 3.30am.
He added that systems vendors were on standby on-site to recover the system within 30 minutes if the trial did not go well.
This time round, the trial caused the storage systems to overload at about 10.40am, and the process was aborted immediately.
“However, the extent of the overload was much more severe than anticipated, and the vendors who were on-site had to work with their global support team to diagnose and reboot the servers,” he said.
Impact of disruption
Assoc Prof Faishal said ICA immediately activated its business continuity plan once the system went down
Off-duty officers were also recalled to help man manual immigration counters and perform crowd control.
Across all the checkpoints, Assoc Prof Faishal said the failover process kicked in, and all the manual counters, and certain automated lanes, switched to backup systems.
However not all the automated lanes have this failover capability, he said, as different models were procured over the years.
At the land checkpoints, Assoc Prof Faishal said travellers experienced delays of up to 30 minutes at the onset of the incident.
Those travelling by car at Woodlands Checkpoint faced another hour as the car arrival zone had to be converted to clear motorbikes manually.
Meanwhile, immigration clearance at the Changi Airport was “significantly impacted”, said Assoc Prof Faishal.
Travellers were re-directed to manual counters for immigration clearance, which saw ICA recalling off-duty officers and deploying administrative staff and retaining the officers from the outgoing shift.
Staff from Changi Airport were also deployed to identify travellers whose flights were departing soon so that their departure clearance could be prioritised.
Despite the measures taken by ICA and Changi Airport, 21 passengers missed their flights at Terminal 4, said Assoc Prof Faishal.
He added that these travellers were subsequently offered to re-book on alternative flights within a week at no extra cost.
Assoc Prof Faishal said that even with the implementation of business continuity plans, degradation in service standards was “inevitable” given the volume of travellers especially at the onset of the incident.
“We apologise to affected travellers for the inconvenience caused and thank them for their understanding,” he said.
Preventing future disruptions
In light of the incident, Assoc Prof Faishal said ICA is reviewing the approach to the upgrade.
“The alternative approach that ICA was initially offered by the vendors was not tenable. It involved upgrading the system directly without replicating the database, but may require five days of system downtime,” he said.
Going forward, Assoc Prof Faishal said ICA is in the process of implementing a multi-year plan, known as the New Clearance Concept (NCC).
This, he said, will transform immigration clearance across all the checkpoints by providing more seamless clearance
He added that it will also help ICA, with limited manpower resources, cope with the ever-rising traveller volume.
Under the new concept, Assoc Prof Faishal said all manual counters and automated lanes will be replaced in phases with the next-generation gates.
This will allow travellers to continue using automated clearance even when the MMBS is down.
In the meantime, Assoc Prof Faishal said the existing automated lanes that have yet to be replaced by the new gates will also be upgraded in the second half of this year.
Once done, he said that Singapore will become one of the first countries in the world to implement NCC-type capabilities.
That said, Assoc Prof Faishal said disruptions will still happen now and then.
“When they do, we will make sure we have robust contingency plans in place, and try to recover as fast as possible. I hope for the public and Members’ understanding,” he said.
The recovery process took about four and a half hours and was back up around 3pm. — TODAY
