IT systems globally are impacted by an operational incident involving a faulty update in CrowdStrike’s Falcon Platform. The issue directly affects Microsoft Windows servers and clients that have the CrowdStrike Falcon agent installed. Impacted systems will crash, restart and experience a Blue Screen of Death (BSOD) unable to load the operating system. This may result in a BSOD reboot cycle. CrowdStrike has ceased deployment of the faulty software, so machines that are not yet affected should not be directly affected moving forward. Several workarounds are provided for impacted devices.

mnemonic’s Incident Response Team (mIRT) is working with affected customers and will continue to provide updates as the situation develops. mnemonic's critical services, including our Security Operations Center (SOC) remain fully operational and were not impacted by this incident.

Current status

CrowdStrike has identified the issue as a bad channel file "C-00000291*.sys" with timestamp 0409 UTC. Files with a timestamp 0527 UTC or later are considered good. CrowdStrike has reverted the deployment of the bad file, so machines that have not yet downloaded the bad file should not be impacted. Affected devices that can successfully reboot and are not stuck in a BSOD reboot cycle should receive the good file as an automatic fix. Machines that are unable to boot will require a workaround to delete the bad file to bring them back online.

The channel file is used by CrowdStrike to continuously deploy updates to detections and critical fixes, and is not a traditional update to the agent software itself. Currently it is unclear for mnemonic what content this specific channel file included.

 

Available workarounds


Workaround for all Windows devices

This workaround needs to be performed manually on every endpoint that requires the workaround.

  1. Boot Windows into Safe Mode or the Windows Recovery Environment (if using BitLocker a recovery key will be required)
  2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
  3. Locate the file matching “C-00000291*.sys”, and delete it.
  4. Boot the host normally.

This workaround can be performed remotely by booting into safe-mode with network, and remotely deleting the files. This may require the user to first login to a VPN. This option will still require a BitLocker recovery key.

Workaround steps for public cloud or similar environments including virtual

Option 1:

  1. Detach the operating system disk volume from the impacted virtual server
  2. Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
  3. Attach/mount the volume to a new virtual server
  4. Navigate to the %WINDIR%\\System32\drivers\CrowdStrike directory
  5. Locate the file matching “C-00000291*.sys”, and delete it.
  6. Detach the volume from the new virtual server
  7. Reattach the fixed volume to the impacted virtual server


Option 2:

Roll back to a snapshot before 0409 UTC.

 

Workaround steps for Azure clients via serial

  1. Login to Azure console --> Go to Virtual Machines --> Select the VM
  2. Upper left on console --> Click : "Connect" --> Click --> Connect --> Click "More ways to Connect" --> Click : "Serial Console"
  3. Once SAC has loaded, type in 'cmd' and press enter.
    1. type in 'cmd' command
    2. type in : ch -si 1
  4. Press any key (space bar). Enter Administrator credentials
  5. Type the following:
    1. bcdedit /set {current} safeboot minimal
    2. bcdedit /set {current} safeboot network
  6. Restart VM
  7. Optional: How to confirm the boot state? Run command:
    wmic COMPUTERSYSTEM GET BootupState


For additional information please see this Microsoft article.