Security researchers at Mandiant say that they have identified a novel new malware ecosystem impacting VMware ESXi security, Linux vCenter servers, and Windows virtual machines – with VMware this week describing it as “persistent and covert” and releasing new mitigation and detection guidance for the techniques outlined in the Mandiant report. (Attackers appear to need to have obtained admin privileges to the ESXi hypervisor.)
The technique involves the use of malicious vSphere Installation Bundles (“VIBs”) to install multiple backdoors on the ESXi hypervisors. The malware lets the attacker maintain persistent admin access to the hypervisor; send commands to the hypervisor that will be executed on guests; send files/commands between the hypervisor and guests as well as guest-to-guest and tamper with logging services on the hypervisor, as well as letting them execute arbitrary commands from one guest VM to another VM on the same hypervisor.
Mandiant researchers said this week that due to the “highly targeted and evasive nature of this intrusion, we suspect motivation to be cyber espionage related” – adding that “we anticipate a variety of other threat actors will use the information outlined in this research to begin building out similar capabilities. Mandiant recommends organizations using ESXi and the VMware infrastructure suite follow the hardening steps… to minimize the attack surface of ESXi hosts. Mandiant’s detailed technical writeup is here (part one) and here (part two).
See: Project Monterey bears hardware-accelerated fruit for VMware
The set of tools was discovered during an incident response. Among several suggested hardening efforts, VMware is encouraging customers to configure Host Acceptance Level to a setting other than “CommunitySupported” (this will require cryptographically signed VIB packages to determine package origin; not a panacea unfortunately) and using Secure Boot, Trusted Platform Modules and its Carbon Black Endpoint.
As Mandiant noted meanwhile, the increased sophisticated of Endpoint Detection and Response (EDR) is little help here: “Certain state-sponsored threat actors have shifted to developing and deploying malware on systems that do not generally support EDR such as network appliances, SAN arrays, and VMware ESXi servers”
VMware added sympathetically in its guidance: “Patching operating systems can be one of the most thankless jobs in IT, but it is the only way to remove a vulnerability from an environment. The ability to migrate running workloads between physical ESXi systems was introduced in 2005 as vMotion, and it revolutionized the ability to keep infrastructure up-to-date independently of workloads. vSphere 7 and 8 each make considerable improvements to vMotion, making migration of workloads seamless again even for large virtual machines.
“Similarly, vSphere Enhanced vMotion Compatibility (EVC) makes vMotion possible across different generations of CPU families, enabling flexibility and futureproofing. vSphere 7 also introduced vSphere Lifecycle Manager, which has a declarative way of managing ESXi hostconfigurations. This includes software configuration, where the introduction of an unauthorized VIB would cause the host to become non-compliant with its configuration baseline. Routine patching enables vSphere Administrators to observe these clues, and restarting ESXi enables Secure Boot to reverify the system configuration to detect malware” the company added in its guidance.
