IT admins were left scrambling to detect if they had been hit by an unknown virus on Friday 13, after Windows applications vanished from the Start menu and taskbar on computers globally following what appears to have been a botched Windows Defender rule triggering chaos for thousands of users globally.
Microsoft first responded responded to the issue at 12:12 BST, saying “we're investigating an issue where users are unable to access application shortcuts on the Start menu and Taskbar in Windows.”
The bug left admins and MSPs left fielding thousands of calls from worried employees and customers after what looks like one of the worst QA failures at Microsoft in some time -- one IT admin noted that "we had this on 'Audit' mode on Security Baseline (Intune) and it still affected the environment..."
Another power user reported that “Desktop and OneDrive items have been moved to OneDrive recyclebin [and the] Start menu and Taskbar items (i.e. system files) seem to be gone.”
UPDATED: Microsoft has published instructions for sysadmins seeking to restore shortcuts that were deleted by the botched update and is advising customers to update to build 1.381.2164.0 or later.
Windows applications vanishing from Start? Buggy rule reverted but...
Microsoft later identified it as the result of a astonishingly bad Friday afternoon code deployment triggering issues, saying “we've reverted the rule to prevent further impact whilst we investigate further.”
Users say it appears to have been “a problem with the newest Defender signature (1.381.2140.0)” with one comment on the /sysadmin Reddit channel that it looks “like that all shortcuts which are located in ProgramData\Microsoft\Windows\Start Menu\Programs will be deleted instantly.”
(As ever Reddit/r/sysadmin is a mine of useful hotfixes and conversation for those affected.)
Microsoft later confirmed that "the revert is in progress and may take several hours to complete. We recommend placing the offending ASR rule into Audit Mode to prevent further impact until the deployment has completed. For more details and instructions, please follow the SI MO497128 in your admin center."
It won't be the only time a Defender update has caused issues.
In December 2021 a new functionality shipped by Microsoft to its own Defender security product set klaxons blaring in SOCs and with security professionals after Defender itself detected it and flagged it as malicious.
The “sensor tampering” alerts are being triggered by Microsoft Defender for Endpoint, seemingly mostly on Windows Server 2016, social media posts suggest, after a new binary, OpenHandleCollector.exe, tries to run.
This appears to have started as early as December 23, with a flurry of users also reporting it triggering alerts on December 29 as Microsoft security investigated. (Users were quick to upload the program to VirusTotal, where no other vendors flagged it as malicious and Microsoft staffers today confirmed it was a false positive.)
Defender flagged sensor tampering after OpenHandleCollector.exe unexpectedly (to Defender) opened a handle to SenseIR processes (C:\program files\Windows Defender Advanced Threat Protection\SenseIR.exe
Closer investigation revealed the process was stemming from Defender’s own legitimate “datacollection” folder.
Quite what Microsoft’s QA team [sic] were thinking is an open question.
A major 2020 Azure outage saw Microsoft explaining that changes “initially target a validation ring that contains no customer data, followed by an inner ring that contains Microsoft only users, and lastly our production environment. These changes are deployed in phases across five rings over several days.
“In this case,” it said at the time, “the SDP [safe deployment process] system failed to correctly target the validation test ring due to a latent defect that impacted the system’s ability to interpret deployment metadata.
"Consequently, all rings were targeted concurrently” with attempts to use automated rollback systems failing because “the latent defect in our SDP system had corrupted the deployment metadata, and we had to resort to manual rollback processes" -- a case that emphasises that computers are hard, but this one was a peach.
