{"id":3295,"date":"2024-09-14T13:40:57","date_gmt":"2024-09-14T12:40:57","guid":{"rendered":"https:\/\/mortenknudsen.net\/?p=3295"},"modified":"2024-09-18T08:00:01","modified_gmt":"2024-09-18T07:00:01","slug":"troubleshooting-monitoring-of-data-collection-rules","status":"publish","type":"post","link":"https:\/\/mortenknudsen.net\/?p=3295","title":{"rendered":"Troubleshooting &amp; Monitoring of Log Ingestion with Data Collection Rules"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">As I have outlined in the series of <a href=\"https:\/\/mortenknudsen.net\/?p=1687\" data-type=\"link\" data-id=\"https:\/\/mortenknudsen.net\/?p=1687\">blogs<\/a>, Azure Logging is based on <strong>Data Collection Rules<\/strong> (DCRs) and <strong>Azure Pipeline<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As log data serves as documentation for compliance and is crucial in SIEM security hunting , it is vital, that we can <strong>monitor<\/strong> the <strong>log ingestion<\/strong> health state and <strong>detect if data is dropped<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This blog covers how we can <strong>monitor the health<\/strong> and <strong>analyze metrics\/performance of the log ingestion<\/strong> using <strong>Data Collection Rules<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"826\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-21-1024x826.png\" alt=\"\" class=\"wp-image-3337\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-21-1024x826.png 1024w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-21-300x242.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-21-768x620.png 768w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-21.png 1389w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Content<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"#implementation\" data-type=\"internal\" data-id=\"#implementation\">Implementation of DCR Diagnostics<\/a><\/li>\n\n\n\n<li><a href=\"#performance\">Performance Monitoring<\/a><\/li>\n\n\n\n<li><a href=\"#health\">Health Monitoring<\/a><\/li>\n\n\n\n<li><a href=\"#monitorperformance\" data-type=\"internal\" data-id=\"#monitorperformance\">Alerting with Azure Monitor (Performance)<\/a><\/li>\n\n\n\n<li><a href=\"#monitorhealth\" data-type=\"internal\" data-id=\"#monitorhealth\">Alerting with Azure Monitor (Health)<\/a><\/li>\n\n\n\n<li><a href=\"#troubleshoot\" data-type=\"internal\" data-id=\"#troubleshoot\">Troubleshooting (Microsoft Article)<\/a><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"implementation\">Implementation of DCR Diagnostics<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In order for us to monitor the ingestion of data through a Data Collection Rule, we need to enable <strong>Diagnostics <\/strong>on the <strong>Data Collection Rule<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"953\" height=\"675\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-7.png\" alt=\"\" class=\"wp-image-3301\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-7.png 953w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-7-300x212.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-7-768x544.png 768w\" sizes=\"auto, (max-width: 953px) 100vw, 953px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1002\" height=\"669\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-6.png\" alt=\"\" class=\"wp-image-3300\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-6.png 1002w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-6-300x200.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-6-768x513.png 768w\" sizes=\"auto, (max-width: 1002px) 100vw, 1002px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Design consideration (cost)<\/h2>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">You don&#8217;t need to enable AllMetrics to see the data using Azure Metrics, as the data is already in the Azure Platform (free).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But if you want to keep the data in logs for compliance purpose (dropped logs) &#8211; or use it for queries, you can enable the AllMetrics above which will stream the data into LogAnalytics which will make the below entries every 1 min. The data is billable, if you stream it into LogAnalytics.<\/p>\n<\/blockquote>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"103\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-23-1024x103.png\" alt=\"\" class=\"wp-image-3350\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-23-1024x103.png 1024w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-23-300x30.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-23-768x77.png 768w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-23-1536x154.png 1536w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-23-2048x205.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"performance\">Performance Monitoring<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Once we have enabled diagnostics, platform metrics data will now be flowing into Azure Monitor (free) so we can analyze a number of metrics.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How do I find the metrics ?<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"447\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-19-1024x447.png\" alt=\"\" class=\"wp-image-3335\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-19-1024x447.png 1024w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-19-300x131.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-19-768x336.png 768w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-19-1536x671.png 1536w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-19.png 1936w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"299\" height=\"180\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-5.png\" alt=\"\" class=\"wp-image-3299\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Now you can select the metric of choice.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"372\" height=\"256\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-2.png\" alt=\"\" class=\"wp-image-3296\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-2.png 372w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-2-300x206.png 300w\" sizes=\"auto, (max-width: 372px) 100vw, 372px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">You can now see the data in the timeline.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"423\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-4-1024x423.png\" alt=\"\" class=\"wp-image-3298\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-4-1024x423.png 1024w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-4-300x124.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-4-768x317.png 768w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-4-1536x635.png 1536w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-4.png 1996w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"health\">Health Monitoring<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Parts of the diagnostics information related to health\/errors goes in the table <strong>DCRLogErrors<\/strong>, which can be queried using Kusto.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"587\" height=\"465\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-8.png\" alt=\"\" class=\"wp-image-3302\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-8.png 587w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-8-300x238.png 300w\" sizes=\"auto, (max-width: 587px) 100vw, 587px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"monitorperformance\">Alerting with Azure Monitor (Performance)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Since we have the data in Azure Monitor, we can also configure alerting using Dynamic Thresholds as shown below. Using machine learning, Azure Monitor will learn the &#8220;normal flow&#8221; so it can detect abnormally dynamically.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"518\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-9-1024x518.png\" alt=\"\" class=\"wp-image-3306\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-9-1024x518.png 1024w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-9-300x152.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-9-768x388.png 768w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-9-1536x777.png 1536w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-9.png 1570w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"monitorhealth\">Alerting with Azure Monitor (Health)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Monitoring of DCR Errors can be done with Azure Monitor custom search query. If number of rows returned after greater than 0, then errors are happening.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1023\" height=\"999\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-20.png\" alt=\"\" class=\"wp-image-3336\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-20.png 1023w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-20-300x293.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-20-768x750.png 768w\" sizes=\"auto, (max-width: 1023px) 100vw, 1023px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"troubleshoot\">Troubleshooting (Microsoft Article)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Microsoft has created a great article or troubleshooting, which can be found <a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/essentials\/data-collection-monitor\" data-type=\"link\" data-id=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/essentials\/data-collection-monitor\">here<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Article covers:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/essentials\/data-collection-monitor#dcr-error-logs\" data-type=\"link\" data-id=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/essentials\/data-collection-monitor#dcr-error-logs\">DCR Error Logs<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/essentials\/data-collection-monitor#dcr-metrics\" data-type=\"link\" data-id=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/essentials\/data-collection-monitor#dcr-metrics\">DCR Metrics<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/essentials\/data-collection-monitor#troubleshooting-common-issues\" data-type=\"link\" data-id=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/essentials\/data-collection-monitor#troubleshooting-common-issues\">Troubleshooting common issues<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/essentials\/data-collection-monitor#monitoring-your-log-ingestion\" data-type=\"link\" data-id=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/essentials\/data-collection-monitor#monitoring-your-log-ingestion\">Monitoring your log ingestion<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/essentials\/data-collection-monitor#alerts\" data-type=\"link\" data-id=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/essentials\/data-collection-monitor#alerts\">Alerts<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>As I have outlined in the series of blogs, Azure Logging is based on Data Collection Rules (DCRs) and Azure &#8230; <\/p>\n<p class=\"read-more-container\"><a title=\"Troubleshooting &amp; Monitoring of Log Ingestion with Data Collection Rules\" class=\"read-more button\" href=\"https:\/\/mortenknudsen.net\/?p=3295#more-3295\" aria-label=\"Read more about Troubleshooting &amp; Monitoring of Log Ingestion with Data Collection Rules\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":3339,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"ngg_post_thumbnail":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[140,55,129,131,128,54,143,141,137,152,142,134,132,58],"tags":[18,5,9,80,17,24],"class_list":["post-3295","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ama","category-azure","category-azure-data-collection-rules","category-azure-data-ingestion-pipeline","category-azure-log-ingestion-api","category-azure-loganalytics","category-azure-logging","category-azure-monitor-agent","category-dcr","category-kusto","category-logging","category-mvpbuzz","category-security","category-sentinel","tag-azure","tag-loganalytics","tag-logging","tag-microsoftsecurity","tag-security","tag-sentinel","infinite-scroll-item","resize-featured-image"],"featured_image_src":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-22.png","author_info":{"display_name":"Morten Knudsen","author_link":"https:\/\/mortenknudsen.net\/?author=1"},"jetpack_featured_media_url":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-22.png","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/3295","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3295"}],"version-history":[{"count":10,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/3295\/revisions"}],"predecessor-version":[{"id":3355,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/3295\/revisions\/3355"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/media\/3339"}],"wp:attachment":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3295"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3295"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3295"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}