A Taiwanese company claimed that withdrawing mis-issued software certificates by an industry-agreed deadline could crash air traffic control, paralyse healthcare facilities and bring down parts of the national grid.
Chunghwa Telecom (which has its own root Certificate Authority, or “CA”) issued 12,911 certs with a “controversial value.” Many of these certificates were used within critical infrastructure and throughout the public sector.
This meant they could not, it claimed controversially, be easily replaced within the five-days stipulated by the CA/Browser Forum Baseline Requirements without triggering nationwide carnage – a case that another CA described as highlighting a widespread industry problem.
Software certificates like TLS/SSL certs are in the spotlight right now after Google announced that Chrome would block sites using certificates issued by large CA Entrust from November 1, 2024. (Such certs are a file installed on a website's origin server containing a public key and other data. Without an SSL certificate, a website's traffic can't be encrypted with TLS.)
Listing the potential damage from certificate revocation, like the grounding of flights, Chunghwa Telecom claimed in a report on Bugzilla “All our users are government agencies, and the government's infrastructure spans across various crucial services nationwide. If we revoke all certificates immediately, the impact will be extensive.”
It went on to warn that revoking the mis-issued SSL certificates would mean “the railway system would be paralysed”, “voltage load monitoring would be paralyzed, making it impossible to manage the national power grid”, and an airport control tower's monitoring system “would be unable to function properly… affecting flight takeoff and landing.”
The ICU centralized monitoring system in the healthcare system would also be “paralyzed, affecting patients' medical rights”, and people would not be able to file their taxes. In a summary of the report, a telecoms company representative said it was unable to withdraw the certificates within a stipulated five-day deadline: "Many user contacts do not have an IT background and cannot replace the certificates themselves,” he said.
“They need to contact IT vendors or equipment suppliers, which usually requires 1-2 weeks to schedule an appointment” he added.
“Government agencies use official documents for approval processes. Replacing certificates requires approval from multiple levels of management, making it very [difficult] to complete within five days… the users… require official documents to be signed and returned…”
The telecoms firm has now revoked all certificates involved in the incident and clarified that certificates used for the information system websites of transportation and logistics departments “were used by passengers to check flight schedules, and not for aviation or its control systems" – which means the situation would not be as bad as it first appeared.
The problem with revoking certificates
We spoke to Nick France, CTO of the CA Sectigo, and asked if revoking certificates could cause the kind of mayhem predicted by the Taiwan telecoms firm/CA.
“It could,” he said. "This has come up over several years with lots of different CAs. These systems are not completely infallible, and mistakes happen. When you make a mistake, you're supposed to document it, detail the incident, remedy the incident, and then answer questions.
“This highlights a general industry problem. Companies are not taking action in the required time frame and are being made to explain why.”
For a CA, revoking a certificate is "actually incredibly easy as a technical process" France told us: “We just basically click a button, and that gets the certificate serial number is then placed on a revocation," he added.
"The problem lies with the customer who has this certificate, which they need to replace. They'll need to go through the whole process to generate a new key and request a certificate, submit that to their CA, receive that certificate, and then go ahead and install it. That process can be done in seconds if you're automated and if you have the right tools in place.
“But what we're seeing are customers of a CA finding it difficult to install a new one, whether they had it installed on a machine that had to have someone physically connect to it, or because of a security issue or an old system surprising that can’t be automated.”
He added: “Certificates used for websites, web apps, or email servers are not designed to be used on a critical national secure system, such as running hospitals or air traffic control. These certificates have to be revoked from time to time, and cycled through. If you have certificates in place where that’s not possible, you’re not using the right tool for the job.”
Certifiable challenges
An inability to quickly withdraw a certificate could cause major security headaches, reducing the speed of incident response.
Kevin Bocek, Chief Innovation Officer at Venafi, also told The Stack: "Like we’ve seen with recent issues involving Entrust, no business or government agency should be unable to quickly replace any certificate. Security teams can’t be held captive or be faced with shutting down based on decisions of their certificate issuer – or the likes of Mozilla and Google.
"This is why experts like Gartner advise security teams to implement effective machine identity management. With proper certificate lifecycle management, security teams can maintain control and quickly replace certificates as needed. On top of this, critical infrastructure systems – like those behind air traffic control and power grids – should avoid using public certificate issuers. Instead, they should run their own PKIaaS to issue certificates. This eliminates third-party risks and dependencies on rules set by entities like Mozilla and Google for online sites, such as retailers and social media platforms.”
Bocek also agreed that the apocalyptic scenario set out during this incident was plausible.
"Revoking certificates in use for air traffic control and power grids could very well stop these systems," he said. "Just like human passports, machine identities – like TLS certificates – have validity periods and can be revoked. Revoking a certificate means connections should no longer be authenticated, causing machines to grind to a halt.
“However, certificates expire and are revoked without incident every day when modern machine identity management is applied. Certificate lifecycle management is now available that makes sure that systems like air traffic control and power grids don’t create outages, like we’ve seen in the last week with CrowdStrike’s failed software update.
"Unfortunately, it seems many of the Taiwanese systems are still running as if it’s 1999 – a surprisingly common issue worldwide, from London to Frankfurt to Taipei. In these places, certificates are often managed using spreadsheets and paper.”i
"What’s most shocking about this issue is that critical systems like air traffic control and power grids don’t have certificate lifecycle management systems – that can quickly replace certificates. The excuse often given is that government approvals and paperwork must be completed. However, in 2024, this approach is no longer viable for achieving the resilience and speed required today. This is especially concerning in Taiwan, where resilience and speed are imperative due to looming Chinese threats.”
We have written to the CA involved in this story for comment.
Chunghwa Telecom faced considerable criticism from CA/Browser members over the incident. It will revoke mis-issued certificates faster next time it said; possibly without claiming the world will end.
“We will add the expiration deadline of the BR [the five-day baseline requirements] to the user agreement, and we have obtained official consent" a staff member noted in the bug report. "This incident has attracted official attention."
