{"id":19748,"date":"2024-07-26T12:37:29","date_gmt":"2024-07-26T12:37:29","guid":{"rendered":"https:\/\/www.lakesidesoftware.com\/?p=19748"},"modified":"2024-07-26T12:37:30","modified_gmt":"2024-07-26T12:37:30","slug":"bugs-happen-why-i-chose-user-mode-for-systracks-agent-design","status":"publish","type":"post","link":"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/","title":{"rendered":"Bugs Happen \u2014 Why I Chose User Mode for SysTrack\u2019s Agent Design\u00a0"},"content":{"rendered":"\n<p>By Mike Schumacher, Lakeside Founder<\/p>\n\n\n\n<p>Last week when I woke up to the fast-spreading news that an IT outage indicated by the dreaded Blue Screen of Death (BSOD) was unfolding, I had two responses at once. 1: Oh, that\u2019s not good. And 2:\u00a0I sure am glad the enterprise digital endpoint monitoring product I designed\u00a0(<a href=\"https:\/\/www.lakesidesoftware.com\/platform\/\" target=\"_blank\" rel=\"noreferrer noopener\">Lakeside SysTrack)<\/a> was not to blame.\u00a0\u00a0<\/p>\n\n\n\n<p>How could I be so certain? It\u2019s simple. When I designed SysTrack nearly 30 years ago, I made sure it would run entirely in user mode, with no <a href=\"https:\/\/learn.microsoft.com\/en-us\/windows-hardware\/drivers\/gettingstarted\/user-mode-and-kernel-mode\" target=\"_blank\" rel=\"noreferrer noopener\">kernel mode<\/a> components. While much of the product has evolved with new innovations over the last three decades, the importance of running in user mode has remained. That means it essentially is running in the sandboxed and safe area provided by the operating system. Recognizing that software bugs happen (even very simple ones) despite arduous testing practices, I wanted the peace of mind in knowing that running our agents in user mode would prevent a CrowdStrike-like scenario from ever happening.&nbsp;&nbsp;&nbsp;<\/p>\n\n\n\n<p>Of course, it is understandable why an endpoint security software like CrowdStrike runs in kernel mode, which is \u201c<a href=\"https:\/\/www.ituonline.com\/tech-definitions\/what-is-kernel-mode\/#:~:text=Definition%3A%20Kernel%20Mode&amp;text=It%20is%20a%20critical%20component,all%20hardware%20and%20system%20resources\" target=\"_blank\" rel=\"noreferrer noopener\">a privileged mode of operation for the central processing unit (CPU) in a computer system\u201d<\/a> that enables greater security given the protected access. The problem with running in kernel mode (as the world now knows), however, is that even a line or two of bad code can take down operating systems.&nbsp;&nbsp;<\/p>\n\n\n\n<p>Last week\u2019s CrowdStrike problem illustrates why not using kernel components, as some of our competitors do, gives Lakeside\u2019s endpoint software customers a big advantage. A bug in a kernel driver, even a trivial bug, can cause a BSOD. Others even acknowledge this risk by trying to build in crash guards that if they crash your system several times, they try to disable themselves. By that time, though, users have been significantly impacted and customers have also been impacted, causing lost time and money.&nbsp; &nbsp;<\/p>\n\n\n\n<p>If the software or app runs entirely in user mode as SysTrack does, by contrast, the software or app could crash but it won\u2019t take down the system &#8230; or worse since kernel components can bypass OS security. The BSOD, however, is the most visible problem \u2014 one that has kept many weary IT teams working around the clock since the July 19 outages hit airlines, banks, and other critical sectors.&nbsp;&nbsp;<\/p>\n\n\n\n<p>Despite some claiming they can automatically fix the CrowdStrike problem, an automation simply is not possible. The remediation requires having to boot a system into Windows safe mode, which must be done manually. Why? Although this article explains the tedium of the recovery process very well, I want to add my own perspective. Certain bugs are triggered before you get far enough in the boot process to interact with the operating system; you have no chance to intervene before you go back into the blue screen. It bears repeating you just don\u2019t have a chance.&nbsp;<\/p>\n\n\n\n<p>Check out this <a href=\"https:\/\/www.reddit.com\/r\/AZURE\/comments\/1e70rdw\/psa_repairing_the_crowdstrike_bsod_on_azurehosted\/?rdt=48215\" target=\"_blank\" rel=\"noreferrer noopener\">thread about the repair process<\/a>. Imagine doing this 50,000 times across an enterprise estate. Having no kernel drivers in SysTrack helps engineers like me sleep better at night. If you ask me, that is an exceptionally small agent footprint; no matter what error we might make, we won\u2019t take the system down.&nbsp;<\/p>\n\n\n\n<p>As I said, because of the nature of CrowdStrike as an endpoint detection and response (EDR) security product and what it does, it must have kernel drivers in it. You can\u2019t provide malware protection without living in the kernel. Unfortunately, with bugs in kernel mode, not only has their program faulted but, as part of the operating systems, their fault took down the whole shooting match with it.&nbsp;<\/p>\n\n\n\n<p>What can you do to avoid this sort of trouble? Vet every software package and don\u2019t accept kernel mode components when they can\u2019t be proven essential. How do you know if there are such drivers in the software? Get a great DEX product (I prefer mine of course, but others are available). Second, don\u2019t have multiple versions of kernel-mode software in your estate; a single version means a smaller risk profile. How do you know which versions you have? Same answer: get a great DEX tool like <a href=\"https:\/\/www.lakesidesoftware.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">SysTrack<\/a>.&nbsp;<\/p>\n\n\n\n<p>Indeed, this incident is an unfortunate one. So what is Lakeside doing to help our own customers affected by the IT outage? Within about 12 hours, our engineers built a dashboard to help our <a href=\"https:\/\/www.lakesidesoftware.com\/blog\/recovering-from-the-crowdstrike-outage\/\" target=\"_blank\" rel=\"noreferrer noopener\">customers recover from the outage<\/a>. Co-created with two affected customers, this dashboard enables our customers to:&nbsp;<\/p>\n\n\n\n<p>1.&nbsp;&nbsp;&nbsp; Understand the magnitude of the impact.&nbsp;<\/p>\n\n\n\n<p>2.&nbsp;&nbsp;&nbsp; Triage repair of high-priority systems.&nbsp;<\/p>\n\n\n\n<p>3.&nbsp;&nbsp;&nbsp; Monitor the progress of remediation at scale.&nbsp;<\/p>\n\n\n\n<p>When it comes to visibility, for instance, our customized dashboard eliminates the need for a time-consuming and costly war-room scenario by giving IT teams data-backed insight into which systems were affected and where they are located. Specifically, the SysTrack dashboard sheds light on the scope and impact of the outage by highlighting how many Windows systems are used across the digital estate, which of those systems have CrowdStrike installed and could be vulnerable, and where those systems are located. In turn, the IT team can prioritize triage efforts and continue to monitor the success of recovery efforts. This prioritization capability is especially important for enterprises that have remote systems that may require boots-on-the-ground fixes.&nbsp;&nbsp;<\/p>\n\n\n\n<p>There\u2019s a saying I like that never gets old for me: \u201cIf you are willing to do only what\u2019s easy, life will be hard. But if you are willing to do what\u2019s hard, life will be easy.\u201d When we first built Lakeside SysTrack, it would have been easier just to build a device driver in kernel mode. Instead, we chose the harder way by designing the agent footprint to run in user-mode.&nbsp;&nbsp;<\/p>\n\n\n\n<p>Of course, any software company can have a bad day if an app fails or crashes, but knowing a catastrophic blue screen never will happen on my watch helps me sleep better. Now, I hope all the red-eyed IT teams working tirelessly since last Friday can soon enjoy what has been elusive sleep interrupted by the kernel bug nightmare.&nbsp;<\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>By Mike Schumacher, Lakeside Founder Last week when I woke up to the fast-spreading news that an IT outage indicated by the dreaded Blue Screen of Death (BSOD) was unfolding, I had two responses at once. 1: Oh, that\u2019s not good. And 2:\u00a0I sure am glad the enterprise digital endpoint monitoring product I designed\u00a0(Lakeside SysTrack)&#8230;<\/p>\n","protected":false},"author":10,"featured_media":19749,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":true,"_kad_blocks_custom_css":"","_kad_blocks_head_custom_js":"","_kad_blocks_body_custom_js":"","_kad_blocks_footer_custom_js":"","_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","footnotes":""},"categories":[515],"tags":[],"class_list":["post-19748","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-it-infrastructure"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.6 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Bugs Happen \u2014 Why I Chose User Mode for SysTrack\u2019s Agent Design\u00a0 | Lakeside Software<\/title>\n<meta name=\"description\" content=\"Last week\u2019s CrowdStrike problem illustrates why not using kernel components, as some of our competitors do, gives Lakeside\u2019s endpoint software customers a big advantage given its agent design for user mode. A bug in a kernel driver, even a trivial bug, can cause a BSOD.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Bugs Happen \u2014 Why I Chose User Mode for SysTrack\u2019s Agent Design\u00a0 | Lakeside Software\" \/>\n<meta property=\"og:description\" content=\"Last week\u2019s CrowdStrike problem illustrates why not using kernel components, as some of our competitors do, gives Lakeside\u2019s endpoint software customers a big advantage given its agent design for user mode. A bug in a kernel driver, even a trivial bug, can cause a BSOD.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/\" \/>\n<meta property=\"og:site_name\" content=\"Lakeside Software\" \/>\n<meta property=\"article:published_time\" content=\"2024-07-26T12:37:29+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-07-26T12:37:30+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.lakesidesoftware.com\/wp-content\/uploads\/2024\/07\/image.jpeg\" \/>\n\t<meta property=\"og:image:width\" content=\"1600\" \/>\n\t<meta property=\"og:image:height\" content=\"1067\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Mike Schumacher\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Mike Schumacher\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Bugs Happen \u2014 Why I Chose User Mode for SysTrack\u2019s Agent Design\u00a0 | Lakeside Software","description":"Last week\u2019s CrowdStrike problem illustrates why not using kernel components, as some of our competitors do, gives Lakeside\u2019s endpoint software customers a big advantage given its agent design for user mode. A bug in a kernel driver, even a trivial bug, can cause a BSOD.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/","og_locale":"en_US","og_type":"article","og_title":"Bugs Happen \u2014 Why I Chose User Mode for SysTrack\u2019s Agent Design\u00a0 | Lakeside Software","og_description":"Last week\u2019s CrowdStrike problem illustrates why not using kernel components, as some of our competitors do, gives Lakeside\u2019s endpoint software customers a big advantage given its agent design for user mode. A bug in a kernel driver, even a trivial bug, can cause a BSOD.","og_url":"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/","og_site_name":"Lakeside Software","article_published_time":"2024-07-26T12:37:29+00:00","article_modified_time":"2024-07-26T12:37:30+00:00","og_image":[{"width":1600,"height":1067,"url":"https:\/\/www.lakesidesoftware.com\/wp-content\/uploads\/2024\/07\/image.jpeg","type":"image\/jpeg"}],"author":"Mike Schumacher","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Mike Schumacher","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/#article","isPartOf":{"@id":"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/"},"author":{"name":"Mike Schumacher","@id":"https:\/\/www.lakesidesoftware.com\/#\/schema\/person\/5869bcdcf96fd1b16ae4c0a602c91194"},"headline":"Bugs Happen \u2014 Why I Chose User Mode for SysTrack\u2019s Agent Design\u00a0","datePublished":"2024-07-26T12:37:29+00:00","dateModified":"2024-07-26T12:37:30+00:00","mainEntityOfPage":{"@id":"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/"},"wordCount":1060,"publisher":{"@id":"https:\/\/www.lakesidesoftware.com\/#organization"},"image":{"@id":"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/#primaryimage"},"thumbnailUrl":"https:\/\/www.lakesidesoftware.com\/wp-content\/uploads\/2024\/07\/image.jpeg","articleSection":["IT Infrastructure"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/","url":"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/","name":"Bugs Happen \u2014 Why I Chose User Mode for SysTrack\u2019s Agent Design\u00a0 | Lakeside Software","isPartOf":{"@id":"https:\/\/www.lakesidesoftware.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/#primaryimage"},"image":{"@id":"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/#primaryimage"},"thumbnailUrl":"https:\/\/www.lakesidesoftware.com\/wp-content\/uploads\/2024\/07\/image.jpeg","datePublished":"2024-07-26T12:37:29+00:00","dateModified":"2024-07-26T12:37:30+00:00","description":"Last week\u2019s CrowdStrike problem illustrates why not using kernel components, as some of our competitors do, gives Lakeside\u2019s endpoint software customers a big advantage given its agent design for user mode. A bug in a kernel driver, even a trivial bug, can cause a BSOD.","breadcrumb":{"@id":"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/#primaryimage","url":"https:\/\/www.lakesidesoftware.com\/wp-content\/uploads\/2024\/07\/image.jpeg","contentUrl":"https:\/\/www.lakesidesoftware.com\/wp-content\/uploads\/2024\/07\/image.jpeg","width":1600,"height":1067},{"@type":"BreadcrumbList","@id":"https:\/\/www.lakesidesoftware.com\/blog\/bugs-happen-why-i-chose-user-mode-for-systracks-agent-design\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.lakesidesoftware.com\/"},{"@type":"ListItem","position":2,"name":"Bugs Happen \u2014 Why I Chose User Mode for SysTrack\u2019s Agent Design\u00a0"}]},{"@type":"WebSite","@id":"https:\/\/www.lakesidesoftware.com\/#website","url":"https:\/\/www.lakesidesoftware.com\/","name":"Lakeside Software","description":"","publisher":{"@id":"https:\/\/www.lakesidesoftware.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.lakesidesoftware.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.lakesidesoftware.com\/#organization","name":"Lakeside Software","url":"https:\/\/www.lakesidesoftware.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.lakesidesoftware.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.lakesidesoftware.com\/wp-content\/uploads\/2024\/01\/Better_View_Knockout-1.png","contentUrl":"https:\/\/www.lakesidesoftware.com\/wp-content\/uploads\/2024\/01\/Better_View_Knockout-1.png","width":2358,"height":811,"caption":"Lakeside Software"},"image":{"@id":"https:\/\/www.lakesidesoftware.com\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.lakesidesoftware.com\/#\/schema\/person\/5869bcdcf96fd1b16ae4c0a602c91194","name":"Mike Schumacher","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/0ae0df43e838aa798f1064d57b2d74834cfb3ab48081044c30e6cbd6bd6fbd92?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/0ae0df43e838aa798f1064d57b2d74834cfb3ab48081044c30e6cbd6bd6fbd92?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/0ae0df43e838aa798f1064d57b2d74834cfb3ab48081044c30e6cbd6bd6fbd92?s=96&d=mm&r=g","caption":"Mike Schumacher"},"url":"https:\/\/www.lakesidesoftware.com\/blog\/author\/mike-schumacher\/"}]}},"taxonomy_info":{"category":[{"value":515,"label":"IT Infrastructure"}]},"featured_image_src_large":["https:\/\/www.lakesidesoftware.com\/wp-content\/uploads\/2024\/07\/image-1024x683.jpeg",1024,683,true],"author_info":{"display_name":"Mike Schumacher","author_link":"https:\/\/www.lakesidesoftware.com\/blog\/author\/mike-schumacher\/"},"comment_info":"","category_info":[{"term_id":515,"name":"IT Infrastructure","slug":"it-infrastructure","term_group":0,"term_taxonomy_id":515,"taxonomy":"category","description":"","parent":0,"count":145,"filter":"raw","cat_ID":515,"category_count":145,"category_description":"","cat_name":"IT Infrastructure","category_nicename":"it-infrastructure","category_parent":0}],"tag_info":false,"_links":{"self":[{"href":"https:\/\/www.lakesidesoftware.com\/wp-json\/wp\/v2\/posts\/19748","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lakesidesoftware.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lakesidesoftware.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lakesidesoftware.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lakesidesoftware.com\/wp-json\/wp\/v2\/comments?post=19748"}],"version-history":[{"count":0,"href":"https:\/\/www.lakesidesoftware.com\/wp-json\/wp\/v2\/posts\/19748\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.lakesidesoftware.com\/wp-json\/wp\/v2\/media\/19749"}],"wp:attachment":[{"href":"https:\/\/www.lakesidesoftware.com\/wp-json\/wp\/v2\/media?parent=19748"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lakesidesoftware.com\/wp-json\/wp\/v2\/categories?post=19748"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lakesidesoftware.com\/wp-json\/wp\/v2\/tags?post=19748"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}