{"id":3163,"date":"2024-07-19T11:11:30","date_gmt":"2024-07-19T10:11:30","guid":{"rendered":"https:\/\/mortenknudsen.net\/?p=3163"},"modified":"2024-07-19T15:19:59","modified_gmt":"2024-07-19T14:19:59","slug":"crowdstrike-issue-5-workarounds","status":"publish","type":"post","link":"https:\/\/mortenknudsen.net\/?p=3163","title":{"rendered":"CrowdStrike issue: Workarounds"},"content":{"rendered":"\n
Info from Microsoft:\nWe've received feedback from customers that several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage.\n\nhttps://azure.status.microsoft\/en-us\/status<\/code><\/pre>\n\n\n\n
From X\/Twitter. Thx to @r3srch3r for summarization.<\/p>\n\n\n\n
Physical machine<\/a><\/h5>\n\n\n\n
physical server<\/a><\/h5>\n\n\n\n
VM on Hyper-V<\/a><\/h5>\n\n\n\n
VM on AWS<\/a><\/h5>\n\n\n\n
VM on Azure<\/a><\/h5>\n\n\n\n
Windows 365<\/a><\/h5>\n\n\n\n
Bitlocker Protected drives<\/a><\/h5>\n\n\n\n
<\/p>\n\n\n\n
Posting for the folks affected by the CrowdStrike BSOD<\/p>\n\n\n\n
Physical machine<\/h2>\n\n\n\n
If you got a physical machine —<\/p>\n\n\n\n
– After 3 failed boots, windows will go into “Automatic Repair” mode. (You might need your bitlocker pin)<\/p>\n\n\n\n
– In the automatic repair page click “Advanced Options” > “Troubleshoot” > “Advanced Options” > “Command Prompt”<\/p>\n\n\n\n
– In this command prompt you can cd to the OS drive and rename the Crowdstrike driver<\/p>\n\n\n\n
““““““““““““““““<\/p>\n\n\n\n
C:<\/p>\n\n\n\n
cd C:\\Windows\\System32\\Drivers\\CrowdStrike<\/p>\n\n\n\n
dir C-00000291*.sys<\/p>\n\n\n\n
ren <filename> <filename_old><\/p>\n\n\n\n
““““““““““““““““<\/p>\n\n\n\n
Locate the file matching \u201cC-00000291*.sys\u201d, and rename it.<\/p>\n\n\n\n
Then exit the command prompt and reboot the machine. Your machine should boot up now<\/p>\n\n\n\n
\u2014\u2014\u2014\u2014\u2014\u2014\u2014<\/p>\n\n\n\n
Posting for CrowdStrike BSOD<\/p>\n\n\n\n
Physical server<\/h2>\n\n\n\n
If you got a physical server where you can detach the hard disk —<\/p>\n\n\n\n
– Setup a new windows machine to use for troubleshooting<\/p>\n\n\n\n
– Detach the hard disk from your broken server and attach it to the new windows machine you’ve setup.<\/p>\n\n\n\n
– Go to diskmgmt.msc and look for the hard disk, Right click and bring it online. If the volume is bitlocker encrypted<\/p>\n\n\n\n
– you will need a recovery key to access the file system (contact your AD admin)  <\/p>\n\n\n\n
– Once you can see the file system<\/p>\n\n\n\n
– Go to  <drive letter>\\Windows\\System32\\Drivers\\CrowdStrike – Locate the file matching \u201cC-00000291*.sys\u201d, and rename it.<\/p>\n\n\n\n
– Then go back to diskmgmt.msc to detach the drive. Attach it back to the server and the machine should boot up now<\/p>\n\n\n\n
\u2014\u2014\u2014\u2014\u2014<\/p>\n\n\n\n
Posting for CrowdStrike BSOD<\/p>\n\n\n\n
VM on Hyper-v<\/h2>\n\n\n\n
If you got a Virtual machine on Hyper-V —<\/p>\n\n\n\n
– Attach a Windows 8\/10 installation iso to the VM. Go to the VM’s settings > Under Hardware > Firmware, Change the boot option to make the iso \/ DVD drive boot first.<\/p>\n\n\n\n
– Now reboot the VM and wait till it gets to the “Install” page. Press “Shift + F10” and this launches a command prompt for you.<\/p>\n\n\n\n
– In the command prompt, run:<\/p>\n\n\n\n
diskpart<\/p>\n\n\n\n
list volume<\/p>\n\n\n\n
exit<\/p>\n\n\n\n
– Locate the drive letter of your windows volume. (The volume label should say “Windows”, you can also check the size to figure it out). Then switch to that drive.<\/p>\n\n\n\n
In the example below, I’ve assumed that it showed Ltr F as the drive with Windows, you should replace F with whatever drive letter you have<\/p>\n\n\n\n
““““““““““““““““<\/p>\n\n\n\n
F:<\/p>\n\n\n\n
cd F:\\Windows\\System32\\Drivers\\CrowdStrike<\/p>\n\n\n\n
dir C-00000291*.sys<\/p>\n\n\n\n
ren <filename> <filename_old><\/p>\n\n\n\n
““““““““““““““““<\/p>\n\n\n\n
Locate the file matching \u201cC-00000291*.sys\u201d, and rename it.<\/p>\n\n\n\n
Then exit the command prompt and detach the iso. Reboot the virtual machine. Your machine should boot up now<\/p>\n\n\n\n
\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014<\/p>\n\n\n\n
Posting for CrowdStrike BSOD<\/p>\n\n\n\n
VM on AWS<\/h2>\n\n\n\n
If you got a VM on AWS —<\/p>\n\n\n\n
You have options to detach the disk from your VM, download it. Modify it. upload it back and swap the OS drive to this.<\/p>\n\n\n\n
or<\/p>\n\n\n\n
You have options to detach the disk from your VM, create a new VM, attach the disk to this new VM as a “data” drive. Modify it. Then detach the data drive and attack it back to the original VM<\/p>\n\n\n\n
The “Modify it” portion remains the same:<\/p>\n\n\n\n
– Go to diskmgmt.msc and look for the hard disk, Right click and bring it online. If the volume is bitlocker encrypted – you will need a recovery key to access the file system (contact your AD admin)<\/p>\n\n\n\n
– Once you can see the file system – Go to<\/p>\n\n\n\n
<drive letter>\\Windows\\System32\\Drivers\\CrowdStrike<\/p>\n\n\n\n
– Locate the file matching \u201cC-00000291*.sys\u201d, and rename it.<\/p>\n\n\n\n
– Then go back to diskmgmt.msc to detach the drive. Attach it back to the original VM and boot up<\/p>\n\n\n\n
\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014<\/p>\n\n\n\n
Posting for the folks affected by the CrowdStrike BSOD<\/p>\n\n\n\n
VM on Azure<\/h2>\n\n\n\n
If you got an Azure VM:<\/p>\n\n\n\n
– Create a very basic Windows VM and upload the image to azure to the same resource group as your broken VM. See:<\/p>\n\n\n\n
lnkd.in\/guCedQk7<\/p>\n\n\n\n
– Stop the VM from the portal. Go to Settings > Disks > Swap OS disk. Point it to the disk you just uploaded and boot up the machine.<\/p>\n\n\n\n
– Attach your original OS disk as a data disk. Now you should be able to go to diskmgmt.msc and look for the hard disk, Right click and bring it online.<\/p>\n\n\n\n
– Once you can see the file system – Go to<\/p>\n\n\n\n
<drive letter>\\Windows\\System32\\Drivers\\CrowdStrike<\/p>\n\n\n\n
– Locate the file matching \u201cC-00000291*.sys\u201d, and rename it.<\/p>\n\n\n\n
– Then go back to diskmgmt.msc to detach the drive.<\/p>\n\n\n\n
– Stop the VM from the portal. Go to Settings > Disks > Detach the data disk. Then click “Swap OS disk”. Point it back to the original OS disk and boot up the machine.<\/p>\n\n\n\n
<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Windows 365<\/h2>\n\n\n\n
On Windows365 you can do a rollback to a time before the incident. I saw a Microsoft person post about this, but can’t find it now.<\/p>\n\n\n\n
<\/p>\n\n\n\n
Bitlocker protected drives (thx to @AttilaBubby<\/a>)<\/h2>\n\n\n\n
\"\"<\/figure>\n","protected":false},"excerpt":{"rendered":"
From X\/Twitter. Thx to @r3srch3r for summarization. Physical machine physical server VM on Hyper-V VM on AWS VM on Azure … <\/p>\n
Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"ngg_post_thumbnail":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-3163","post","type-post","status-publish","format-standard","hentry","category-uncategorized","infinite-scroll-item"],"featured_image_src":null,"author_info":{"display_name":"Morten Knudsen","author_link":"https:\/\/mortenknudsen.net\/?author=1"},"jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/3163","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3163"}],"version-history":[{"count":5,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/3163\/revisions"}],"predecessor-version":[{"id":3172,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/3163\/revisions\/3172"}],"wp:attachment":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3163"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3163"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3163"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}