One of the medical applications hit during a ransomware attack on Advanced will not be back online until the end of this year at best – despite backups being available as recently as the day before the incident – the Managed Services Provider (MSP) has now revealed, saying it will need to “fully rebuild and reconfigure the Staffplan environment”, as documents seen by The Stack reveal the extent of potential risk for users.
Advanced revealed a ransomware attack on August 5 that took out seven different widely used healthcare applications it owns and hosts, including one, “Adastra” that is used by the NHS’s emergency 111 hotline.
Staffplan is used by over 79,000 care home workers and a hazard log spreadsheet for the software shows that the risks of server disruption include “medication doses missed”, “basic needs not met, such as nutrition and personal care”, “health needs not met, such as wound care and physical support” and “required number of carers not met” as nightmare service restoration efforts continue for the MSP in the wake of the incident.
Sources close to the company said the application was written in Delphi 5, a development tool for Windows released in 1999 and had not been "significantly" updated since it was written in 2009. It is likely to have a range of complex dependencies and libraries that require legacy versions of Windows to function.
Restoration of Adastra has been a priority and progressed faster, although not all customers are back online yet. The Staffplan rebuild delays were blamed by Advanced on September 13 on “additional complexities uncovered by our forensic experts [that mean] much of the hardware needed to rebuild Staffplan has had to remain under investigation for longer than we anticipated… so that when we rebuild the system in a new, clean environment, it is brought online in a secure state, and our customers can feel confident that it is safe…”
“We anticipate having a Staffplan Minimum Viable Product (i.e. File Servers, SQL, Backups and Citrix) operational by 30 September. Our aim is that subsequent phases, including mapping to our customers environments and returning Staffplan to full restoration will be completed by the end of 2022” it said, in a blow to service users.
It had earlier suggested that contingency plans would only be in place for four weeks.
That means a rebuild lasting over five months, despite the September 13 update confirming that “data is available up to the date of our last backup on 3 August – the day before the cyber-attack occurred.”
Commercial documents on Advanced’s approach to resilience reveal StaffPlan data was “synchronously replicated between the two [active/active data centres] using Microsoft and web technologies including Hyper-V Server virtualisation and SQL Availability Groups” to provide “operational continuity in the unlikely event of a partial or complete failure of either centre. Communication lines between the data centres, N3 and the Internet are resilient, with diverse points of presence [and] backed-up weekly using Dell DR4100 Disk Backup Appliances among other safeguards” an earlier G-Cloud (government procurement) document showed.
Independent researcher Rob Dyke told The Stack that he understood the application in question was written in the ageing Delphi 5 language. (We could not immediately confirm this.)
Recent documentation seen by The Stack suggests that it only removed a dependency on the legacy and now discontinued Microsoft Silverlight application framework earlier this year as well (“This release completes work begun in version 21.1, on the modernisation of the Exchange platform. It removes the Microsoft Silverlight dependency, which has reached end-of-life” the February 2022-dated documents show.)
He said: “If you've got a Delphi 5, you need runtimes that will only work on certain versions of Windows, that you can only then work on certain versions of virtualization. So it could well be that we're looking at quite an old bug in the hypervisor, or in a VPN gateway or Citrix gateway, that was the ingress for the attack. Maybe they don't have the skills or the capacity to be able to rebuild so much so quickly.”
Dyke, who has worked in the medical software environment, added: “I think that they have under-invested in it, under-documented it, and probably been frustrated by the age of the application…”
Security expert James Bore added: “The timeline looks to me like serious, fundamental flaws in their security regime that'll mean if they stand things straight back up they'll immediately be vulnerable again. Four months is a tight deadline to rearchitect a system securely and rebuild your entire security governance program…”
Information security specialist, Jessica Tubbs, added to The Stack, whilst emphasising that her comments were speculative without further details: “[It sounds like they need to] plan, design, build, and deploy an entirely new infrastructure which would likely be incompatible with the backups created by the compromised infrastructure, so they would also have to plan, design, and implement a data migration and conversion for restoration. then there would be testing and validating before… [returning to] full production status.”
Advanced has declined to comment in further detail on the breach beyond its regular updates on each application, saying it is waiting for the outcome of an investigation by the Information Commissioner’s Office.
