{"id":3308,"date":"2024-09-14T12:45:22","date_gmt":"2024-09-14T11:45:22","guid":{"rendered":"https:\/\/mortenknudsen.net\/?p=3308"},"modified":"2024-09-18T07:46:03","modified_gmt":"2024-09-18T06:46:03","slug":"optimize-costs-using-auxiliary-logs-for-verbose-logging","status":"publish","type":"post","link":"https:\/\/mortenknudsen.net\/?p=3308","title":{"rendered":"Optimize Costs using Auxiliary Logs for Verbose Logging"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Today, we use logging for many purposes including security hunting with SIEM (Sentinel), troubleshooting, performance telemetry, compliance reporting &#8211; but it can also be very costly. In my blog below, I will show you how I reduced my log-costs with <strong>89%<\/strong> of the costs (ingestion &amp; query) &#8211; with <strong>96% cost reduction on log ingestion<\/strong> alone.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As mentioned, one of the challenges, I meet is the <strong>costs<\/strong> with the <strong>Analytics plan<\/strong> in LogAnalytics for some use-cases with high ingestion and low retention needs. Recently, Microsoft released the newest in class; <strong>Auxiliary<\/strong> plan, which I find fits perfectly into many scenarios. This blog will cover cost comparisons, implementation and use-cases.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"112\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-10-1024x112.png\" alt=\"\" class=\"wp-image-3309\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-10-1024x112.png 1024w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-10-300x33.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-10-768x84.png 768w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-10.png 1370w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Content<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"#usecase\" data-type=\"internal\" data-id=\"#usecase\">Use-cases for Auxiliary logs<\/a><\/li>\n\n\n\n<li><a href=\"#switch\" data-type=\"internal\" data-id=\"#switch\">Can I switch my Syslog or SecurityEvent table to Auxiliary logs ?<\/a><\/li>\n\n\n\n<li><a href=\"#featurecomparison\" data-type=\"internal\" data-id=\"#featurecomparison\">Feature Comparison of Plans<\/a><\/li>\n\n\n\n<li><a href=\"#costcomparison\" data-type=\"internal\" data-id=\"#costcomparison\">Cost Comparison of Plans<\/a><\/li>\n\n\n\n<li><a href=\"#implementation\" data-type=\"internal\" data-id=\"#implementation\">Implementation of Auxiliary plan<\/a><\/li>\n\n\n\n<li><a href=\"#limitations\">Public Preview limitations<\/a><\/li>\n\n\n\n<li><a href=\"#moreinfo\">More Info<\/a><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"usecase\">Use-cases for Auxiliary logs<\/h2>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">Auxiliary logs can be used to collect any <strong>custom source of data<\/strong> that will be sent into <strong>custom logs<\/strong> using <strong>Data Collection Rule<\/strong>. All data can be queried using Kusto. Data can be queried interactively for 30 days and using search-jobs can be queried up to 12 years.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Use-cases includes <strong>high ingestion (verbose-logging)<\/strong> cases like <strong>storage access logs, NetFlow logs, proxy logs, IoT logs, Firewall logs<\/strong>, <strong>CSV-files, TXT-files<\/strong>, etc.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This can also include compliance use-cases where we need to store the logs for long-term usage.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">See comparison and detailed feature overview later in the blog.<\/p>\n<\/blockquote>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"345\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-17-1024x345.png\" alt=\"\" class=\"wp-image-3325\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-17-1024x345.png 1024w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-17-300x101.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-17-768x259.png 768w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-17.png 1278w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"switch\">Can I switch my Syslog or SecurityEvent table to Auxiliary logs ?<\/h2>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">Currently, standard streams like <strong>Microsoft-CommonSecurityLog<\/strong> (Syslog\/CEF), <strong>Microsoft-SecurityEvent<\/strong> (Windows Security Logs) and <strong>Microsoft-ServiceMap<\/strong> (VM Insights) are not supported to be stored in Auxiliary plan. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">They require to be stored in Analytics plans. The advantage of this is, that Microsoft maintains the schema of the table, which will ensure hunting capabilities in Sentinel can continue to evolve over time when Microsoft continues to enrich the data.<\/p>\n<\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"featurecomparison\">Feature Comparison of Plans<\/h2>\n\n\n\n<figure class=\"wp-block-table alignwide has-small-font-size\"><table class=\"has-fixed-layout\"><tbody><tr><td><\/td><td><strong>Analytics<\/strong> plan<\/td><td><strong>Basic <\/strong>plan<\/td><td><strong>Auxiliary<\/strong> plan<\/td><\/tr><tr><td>Retention (short-term)<\/td><td>31 days (90 days with Sentinel)<\/td><td>30 days<\/td><td>30 days<\/td><\/tr><tr><td>Retention (long-term)<\/td><td>12 years<\/td><td>12 years<\/td><td>12 years<\/td><\/tr><tr><td>Query (interactive)<\/td><td>Up to 2 years<\/td><td>30 days<\/td><td>30 days<\/td><\/tr><tr><td>Query older data<\/td><td>Search jobs<\/td><td>Search jobs<\/td><td>Search jobs<\/td><\/tr><tr><td>Performance (interactive)<\/td><td>High<\/td><td>High<\/td><td>Medium (10-20% slower)<\/td><\/tr><tr><td>Query<\/td><td>Fast query performance with full KQL support<\/td><td>Fast query performance with full KQL on a single table and lookup to Analytics tables<\/td><td>Full KQL on a single table and lookup to Analytics tables<\/td><\/tr><tr><td>Log Ingestion Cost<\/td><td>$2.99&nbsp;per GB<\/td><td>$0.65&nbsp;per GB<\/td><td>$0.13&nbsp;per GB<\/td><\/tr><tr><td>Query Cost<\/td><td>Included<\/td><td>$0.0065&nbsp;per GB of data scanned<\/td><td>$0.0065&nbsp;per GB of data scanned<\/td><\/tr><tr><td>Sentinel Query Support<br>Query cost will apply per query for Basic &amp; Auxiliary plans<\/td><td>Included<\/td><td>Included<\/td><td>Included<\/td><\/tr><tr><td><a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/logs\/summary-rules?tabs=api\" data-type=\"link\" data-id=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/logs\/summary-rules?tabs=api\">Summary Rules support<\/a><\/td><td>Included<\/td><td>Included<\/td><td>Included<\/td><\/tr><tr><td>Alerting capability<br><br>Query cost will apply per query for Basic &amp; Auxiliary plans<br><br>Azure Monitor cost will apply for all alerts.<\/td><td>Included<\/td><td>Included<\/td><td>Included<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">See Public Preview limitations of Auxiliary plan later in this blog (<a href=\"#limitations\" data-type=\"internal\" data-id=\"#limitations\">link<\/a>)<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"costcomparison\">Cost Comparison of Plans<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Below you will find the comparison of the costs between LogAnalytics  and Auxiliary plan. as you can see, I save <strong>89%<\/strong> using Auxiliary plan compared with Analytics plan (based on the assumptions). <\/p>\n\n\n\n<figure class=\"wp-block-table alignwide has-small-font-size\"><table class=\"has-fixed-layout\"><tbody><tr><td>Plan<\/td><td>Action<\/td><td class=\"has-text-align-right\" data-align=\"right\">Amount (Gb)<\/td><td class=\"has-text-align-center\" data-align=\"center\">Days\/Month<\/td><td class=\"has-text-align-right\" data-align=\"right\">USD\/Gb<\/td><td class=\"has-text-align-right\" data-align=\"right\">USD\/Month<\/td><td class=\"has-text-align-right\" data-align=\"right\">Savings % (compared to LogAnalytics plan)<\/td><\/tr><tr><td>ANALYTICS<\/td><td>Ingestion<\/td><td class=\"has-text-align-right\" data-align=\"right\">100<\/td><td class=\"has-text-align-center\" data-align=\"center\">30,7<\/td><td class=\"has-text-align-right\" data-align=\"right\">2,99<\/td><td class=\"has-text-align-right\" data-align=\"right\">9179<\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><\/tr><tr><td><\/td><td>Query<\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><td class=\"has-text-align-center\" data-align=\"center\"><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><td class=\"has-text-align-right\" data-align=\"right\">0<\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><\/tr><tr><td><\/td><td><strong>TOTAL (USD\/Month)<\/strong><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><td class=\"has-text-align-center\" data-align=\"center\"><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><td class=\"has-text-align-right\" data-align=\"right\"><strong>9179<\/strong><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><\/tr><tr><td><\/td><td><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><td class=\"has-text-align-center\" data-align=\"center\"><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><\/tr><tr><td>BASIC<\/td><td>Ingestion<\/td><td class=\"has-text-align-right\" data-align=\"right\">100<\/td><td class=\"has-text-align-center\" data-align=\"center\">30,7<\/td><td class=\"has-text-align-right\" data-align=\"right\">0,65<\/td><td class=\"has-text-align-right\" data-align=\"right\">1996<\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><\/tr><tr><td><\/td><td>Query<br><br># of queries<br><br>Gb scanned per query<\/td><td class=\"has-text-align-right\" data-align=\"right\">100<br><br><br>1000<\/td><td class=\"has-text-align-center\" data-align=\"center\"><\/td><td class=\"has-text-align-right\" data-align=\"right\">0,0065<\/td><td class=\"has-text-align-right\" data-align=\"right\">650<\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><\/tr><tr><td><\/td><td><strong>TOTAL (USD\/Month)<\/strong><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><td class=\"has-text-align-center\" data-align=\"center\"><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><td class=\"has-text-align-right\" data-align=\"right\"><strong>2646<\/strong><\/td><td class=\"has-text-align-right\" data-align=\"right\"><strong>72%<\/strong><\/td><\/tr><tr><td><\/td><td><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><td class=\"has-text-align-center\" data-align=\"center\"><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><\/tr><tr><td>AUXILIARY<\/td><td>Ingestion<\/td><td class=\"has-text-align-right\" data-align=\"right\">100<\/td><td class=\"has-text-align-center\" data-align=\"center\">30,7<\/td><td class=\"has-text-align-right\" data-align=\"right\">0,13<\/td><td class=\"has-text-align-right\" data-align=\"right\">399<\/td><td class=\"has-text-align-right\" data-align=\"right\"><strong>96%<\/strong><\/td><\/tr><tr><td><\/td><td>Query<br><br># of queries<br><br>Gb scanned per query<\/td><td class=\"has-text-align-right\" data-align=\"right\"><br>100<br><br>1000<\/td><td class=\"has-text-align-center\" data-align=\"center\"><\/td><td class=\"has-text-align-right\" data-align=\"right\">0,0065<\/td><td class=\"has-text-align-right\" data-align=\"right\">650<\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><\/tr><tr><td><\/td><td><strong>TOTAL (USD\/Month)<\/strong><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><td class=\"has-text-align-center\" data-align=\"center\"><\/td><td class=\"has-text-align-right\" data-align=\"right\"><\/td><td class=\"has-text-align-right\" data-align=\"right\"><strong>1049<\/strong><\/td><td class=\"has-text-align-right\" data-align=\"right\"><strong>89%<\/strong><\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">Pricing can be found <a href=\"https:\/\/azure.microsoft.com\/en-us\/pricing\/details\/monitor\/\">here<\/a> <\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"implementation\">Implementation of Auxiliary Logs<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In this blog, I have documented, how we can collect the file DBError.txt in the folder C:\\Windows\\System32\\CatRoot2 and look for any Error -1811, which indicates the server was impacted by the August 2024 Windows Update issue covered in this <a href=\"https:\/\/mortenknudsen.net\/?p=3200\" data-type=\"link\" data-id=\"https:\/\/mortenknudsen.net\/?p=3200\">article<\/a>. The data will be stored in Auxiliary table.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The sample involves collection of content from a TXT file so the steps to send data into Auxiliary table are:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Deploy a Data Collection Endpoint in the region where the LogAnalytics workspace is placed. <\/li>\n\n\n\n<li>Create the Auxiliary table with required schema<\/li>\n\n\n\n<li>Create the DCR<\/li>\n\n\n\n<li>Modify the DCR to extend with 2 extra properties so we know from which VM the data in the TXT-file is coming from<\/li>\n\n\n\n<li>Send data in using Log Ingestion API &#8211; or by associating the DCR to VMs in Azure.<\/li>\n<\/ol>\n\n\n\n<h4 class=\"wp-block-heading\">Detailed steps<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">1. Deploy a Data Collection Endpoint in the region where the LogAnalytics workspace is placed. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">See <a href=\"https:\/\/mortenknudsen.net\/?p=1442\">this article<\/a> for instructions on how to create a Data Collection Endpoint.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">2. Pre-create the Auxiliary table with required schema. <\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>In this sample, I create the table <strong>CatRoot2ErrorsAuxiliary_CL<\/strong><\/li>\n<\/ul>\n\n\n\n<ol class=\"wp-block-list\">\n<li><\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code># Connect to Azure\nConnect-AzAccount\n\n\n# Create table CatRoot2ErrorsAuxiliaryV2_CL\n\n$PayLoad = @{\n    properties = @{\n        totalRetentionInDays = 365\n        plan = \"Auxiliary\"\n        schema = @{\n            name = \"CatRoot2ErrorsAuxiliaryV2_CL\"\n            columns = @(\n                        @{\n                            name = \"TimeGenerated\"\n                            type = \"DateTime\"\n                         }\n                        @{\n                            name = \"RawData\"\n                            type = \"String\"\n                         }\n                        @{\n                            name = \"Computer\"\n                            type = \"String\"\n                         }\n                        )\n        }\n    }\n            }\n$PayLoadJson = $PayLoad | ConvertTo-Json -Depth 20\n\n$URI = \"\/subscriptions\/xxxxxx\/resourcegroups\/rg-loganalytics\/providers\/microsoft.operationalinsights\/workspaces\/log-platform-management-errors-p\/tables\/CatRoot2ErrorsAuxiliaryV2_CL?api-version=2023-01-01-preview\"\n\nInvoke-AzRestMethod -Path $URI -Method PUT -Payload $PayLoadJson<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Note the structure in Auxiliary plan is almost the same as an Analytics Plan. Only difference is shown below<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>                                     totalRetentionInDays = 365\n                                     plan = \"Auxiliary\"<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">3. Setup the collection of data &#8211; either using Azure Monitor Agent or Log Ingestion API. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Below is a sample of collecting a text log-file. It is same method using Auxiliary and Analytics<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"436\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-12-1024x436.png\" alt=\"\" class=\"wp-image-3312\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-12-1024x436.png 1024w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-12-300x128.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-12-768x327.png 768w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-12.png 1201w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"726\" height=\"454\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-13.png\" alt=\"\" class=\"wp-image-3313\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-13.png 726w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-13-300x188.png 300w\" sizes=\"auto, (max-width: 726px) 100vw, 726px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"970\" height=\"348\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-14.png\" alt=\"\" class=\"wp-image-3314\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-14.png 970w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-14-300x108.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-14-768x276.png 768w\" sizes=\"auto, (max-width: 970px) 100vw, 970px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">4. Modify the DCR to extend the collection with <strong>Computer<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once you have created the DCR, we can see in the JSON format the below structure, but we need to add the built-in property: <strong>Computer<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code>{\n    \"properties\": {\n        \"immutableId\": \"dcr-e1fxxxxxxxx\",\n        \"dataCollectionEndpointId\": \"\/subscriptions\/xxxxxxxx-3f904de63b79\/resourceGroups\/rg-loganalytics\/providers\/Microsoft.Insights\/dataCollectionEndpoints\/dce-log-platform-management-operation-neu-p\",\n        \"streamDeclarations\": {\n            \"Custom-Text-CatRoot2ErrorsAuxiliary_CL\": {\n                \"columns\": &#91;\n<strong>                    {\n                        \"name\": \"TimeGenerated\",\n                        \"type\": \"datetime\"\n                    },\n                    {\n                        \"name\": \"RawData\",\n                        \"type\": \"string\"<\/strong>\n                    }\n                ]\n            }\n        },\n        \"dataSources\": {\n            \"logFiles\": &#91;\n                {\n                    \"streams\": &#91;\n                        \"Custom-Text-CatRoot2ErrorsAuxiliary_CL\"\n                    ],\n                    \"filePatterns\": &#91;\n                        \"c:\\\\windows\\\\system32\\\\catroot2\\\\dberr.txt\"\n                    ],\n                    \"format\": \"text\",\n                    \"settings\": {\n                        \"text\": {\n                            \"recordStartTimestampFormat\": \"ISO 8601\"\n                        }\n                    },\n                    \"name\": \"Custom-Text-CatRoot2ErrorsAuxili\"\n                }\n            ]\n        },\n        \"destinations\": {\n            \"logAnalytics\": &#91;\n                {\n                    \"workspaceResourceId\": \"\/subscriptions\/xxxxxx-3f904de63b79\/resourceGroups\/rg-loganalytics\/providers\/Microsoft.OperationalInsights\/workspaces\/log-platform-management-errors-p\",\n                    \"workspaceId\": \"xxxxx\",\n                    \"name\": \"la-2145994651\"\n                }\n            ]\n        },\n        \"dataFlows\": &#91;\n            {\n                \"streams\": &#91;\n                    \"Custom-Text-CatRoot2ErrorsAuxiliary_CL\"\n                ],\n                \"destinations\": &#91;\n                    \"la-2145994651\"\n                ],\n                \"outputStream\": \"Custom-CatRoot2ErrorsAuxiliary_CL\"\n            }\n        ],\n        \"provisioningState\": \"Succeeded\"\n    },\n    \"location\": \"northeurope\",\n    \"tags\": {},\n    \"kind\": \"Windows\",\n    \"id\": \"\/subscriptions\/xxxxxxxxx\/resourceGroups\/rg-loganalytics\/providers\/Microsoft.Insights\/dataCollectionRules\/dcr-Operation-CatRoot2ErrorsAuxiliary\",\n    \"name\": \"dcr-Operation-CatRoot2ErrorsAuxiliary\",\n    \"type\": \"Microsoft.Insights\/dataCollectionRules\",\n    \"etag\": \"\\\"5a00e210-0000-0c00-0000-66e0bce80000\\\"\",\n    \"systemData\": {\n        \"createdBy\": \"mok@2linkit.net\",\n        \"createdByType\": \"User\",\n        \"createdAt\": \"2024-09-10T21:28:42.806711Z\",\n        \"lastModifiedBy\": \"mok@2linkit.net\",\n        \"lastModifiedByType\": \"User\",\n        \"lastModifiedAt\": \"2024-09-10T21:40:55.2792351Z\"\n    }\n}<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In order to add the 2 properties, we will do the following 3 steps:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Retrieve the entire DCR using REST API (GET) in JSON format \u2013 and save it to a TXT file<\/li>\n\n\n\n<li>Edit the file \u2013 and add the&nbsp;property <strong>Computer<\/strong> parameter in the&nbsp;<strong>streamDeclarations<\/strong> section<\/li>\n\n\n\n<li>Upload the entire file content using REST API (PUT)<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">4.1 Get the DCR &#8211; save to local JSON file<\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code># here you put the ResourceID of the Data Collection Rules (a sample is provided below)\n$ResourceId = \"\/subscriptions\/7e867037-59b5-4edc-97ee-3f904de63b79\/resourceGroups\/rg-loganalytics\/providers\/Microsoft.Insights\/dataCollectionRules\/dcr-Operation-CatRoot2ErrorsAuxiliary\"\n\n    \n# here you put a path and file name where you want to store the temporary file-extract from DCR (a sample is provided below)\n$FilePath   = \"c:\\tmp\\dcr-export.txt\"\n\n\n####################################################\n# Get DCR\n####################################################\n\n$DCR = Invoke-AzRestMethod -Path (\"$ResourceId\"+\"?api-version=2022-06-01\") -Method GET\n\n$DCR.Content | ConvertFrom-Json | ConvertTo-Json -Depth 20 | Out-File -FilePath $FilePath<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">4.2 Add these lines in the streamDeclarations section<\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code>\"streamDeclarations\":  {\n                            \"Custom-Text-CatRoot2ErrorsAuxiliary_CL\":  {\n                                                                            \"columns\":  &#91;\n                                                                                            {\n                                                                                                \"name\":  \"TimeGenerated\",\n                                                                                                \"type\":  \"datetime\"\n                                                                                            },\n                                                                                            {\n                                                                                                \"name\":  \"RawData\",\n                                                                                                \"type\":  \"string\"\n                                                                                            }<strong>,<\/strong>\n<strong>                                                                                            {\n                                                                                                \"name\":  \"Computer\",\n                                                                                                \"type\":  \"string\"\n                                                                                            }\n<\/strong>                                                                                        ]\n                                                                        }\n                        },\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Don&#8217;t forget the , (comma) after the prior } so the syntax of the JSON file doesn&#8217;t break !<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">4.3 Upload the modified JSON and overwrite DCR using PUT REST Api<\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code>####################################################\n# UPLOAD FILE \/ UPDATE DCR\n####################################################\n\n$DCRContent = Get-Content $FilePath -Raw \n\nInvoke-AzRestMethod -Path (\"$ResourceId\"+\"?api-version=2022-06-01\") -Method PUT -Payload $DCRContent<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Your DCR will now look like this in JSON view<\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code>{\n    \"properties\": {\n        \"immutableId\": \"dcr-exxxxxxxxxf50\",\n        \"dataCollectionEndpointId\": \"\/subscriptions\/xxxxxxxxxxx3b79\/resourceGroups\/rg-loganalytics\/providers\/Microsoft.Insights\/dataCollectionEndpoints\/dce-log-platform-management-operation-neu-p\",\n        \"streamDeclarations\": {\n            \"Custom-Text-CatRoot2ErrorsAuxiliary_CL\": {\n                \"columns\": &#91;\n                    {\n                        \"name\": \"TimeGenerated\",\n                        \"type\": \"datetime\"\n                    },\n                    {\n                        \"name\": \"RawData\",\n                        \"type\": \"string\"\n                    },\n<strong>                    {\n                        \"name\": \"Computer\",\n                        \"type\": \"string\"\n                    }<\/strong>\n                ]\n            }\n        },\n        \"dataSources\": {\n            \"logFiles\": &#91;\n                {\n                    \"streams\": &#91;\n                        \"Custom-Text-CatRoot2ErrorsAuxiliary_CL\"\n                    ],\n                    \"filePatterns\": &#91;\n                        \"c:\\\\windows\\\\system32\\\\catroot2\\\\dberr.txt\"\n                    ],\n                    \"format\": \"text\",\n                    \"settings\": {\n                        \"text\": {\n                            \"recordStartTimestampFormat\": \"ISO 8601\"\n                        }\n                    },\n                    \"name\": \"Custom-Text-CatRoot2ErrorsAuxili\"\n                }\n            ]\n        },\n        \"destinations\": {\n            \"logAnalytics\": &#91;\n                {\n                    \"workspaceResourceId\": \"\/subscriptions\/xxxxxxxxxb79\/resourceGroups\/rg-loganalytics\/providers\/Microsoft.OperationalInsights\/workspaces\/log-platform-management-errors-p\",\n                    \"workspaceId\": \"58c7aa3e-a321-4fdb-99cf-d1096b0f0251\",\n                    \"name\": \"la-2145994651\"\n                }\n            ]\n        },\n        \"dataFlows\": &#91;\n            {\n                \"streams\": &#91;\n                    \"Custom-Text-CatRoot2ErrorsAuxiliary_CL\"\n                ],\n                \"destinations\": &#91;\n                    \"la-2145994651\"\n                ],\n                \"outputStream\": \"Custom-CatRoot2ErrorsAuxiliary_CL\"\n            }\n        ],\n        \"provisioningState\": \"Succeeded\"\n    },\n    \"location\": \"northeurope\",\n    \"tags\": {},\n    \"kind\": \"Windows\",\n    \"id\": \"\/subscriptions\/xxxxxxxxxxxx\/resourceGroups\/rg-loganalytics\/providers\/Microsoft.Insights\/dataCollectionRules\/dcr-Operation-CatRoot2ErrorsAuxiliary\",\n    \"name\": \"dcr-Operation-CatRoot2ErrorsAuxiliary\",\n    \"type\": \"Microsoft.Insights\/dataCollectionRules\",\n    \"etag\": \"\\\"ca00062f-0000-0c00-0000-66e566df0000\\\"\",\n    \"systemData\": {\n        \"createdBy\": \"mok@2linkit.net\",\n        \"createdByType\": \"User\",\n        \"createdAt\": \"2024-09-10T21:28:42.806711Z\",\n        \"lastModifiedBy\": \"mok@2linkit.net\",\n        \"lastModifiedByType\": \"User\",\n        \"lastModifiedAt\": \"2024-09-14T10:35:10.2224854Z\"\n    }\n}<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">5. You are now almost good to go, now you just need to associate VMs, where you want to collect the content of the file and send it into the Auxiliary log table<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"310\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-15-1024x310.png\" alt=\"\" class=\"wp-image-3315\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-15-1024x310.png 1024w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-15-300x91.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-15-768x232.png 768w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-15.png 1240w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">NOTE:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>It may take 10-15 min before the Pipeline is active and data will be ready to flow.<\/li>\n\n\n\n<li>Remember you need to do some changes in the file (DBerror.txt) &#8211; or for example restart the VM before new data will come in, as it doesn&#8217;t take the prior content; only new.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"limitations\">Public preview limitations<\/h2>\n\n\n\n<h4 class=\"wp-block-heading\">Region<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Americas<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Canada Central<\/li>\n\n\n\n<li>Central US<\/li>\n\n\n\n<li>East US<\/li>\n\n\n\n<li>East US 2<\/li>\n\n\n\n<li>West US<\/li>\n\n\n\n<li>South Central US<\/li>\n\n\n\n<li>North Central US<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Asia Pacific<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Australia East<\/li>\n\n\n\n<li>Australia South East<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Europe<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>East Asia<\/li>\n\n\n\n<li>North Europe<\/li>\n\n\n\n<li>UK South<\/li>\n\n\n\n<li>Germany West Central<\/li>\n\n\n\n<li>Switzerland North<\/li>\n\n\n\n<li>France Central<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Middle East<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Israel Central<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Important info about Tables with the Auxiliary plan<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Are currently unbilled. There&#8217;s currently no charge for ingestion, queries, search jobs, and long-term retention.<\/li>\n\n\n\n<li>Do not support columns with dynamic data.<\/li>\n\n\n\n<li>Have a fixed total retention of 365 days.<\/li>\n\n\n\n<li>Support ISO 8601 datetime format only.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Limitations for Data Collection Rules that sends data to a table with an Auxiliary plan<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Can only send data to a single table.<\/li>\n\n\n\n<li>Can&#8217;t include a&nbsp;<a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/essentials\/data-collection-transformations\">transformation<\/a>.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"moreinfo\">More info<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/logs\/create-custom-table-auxiliary\" data-type=\"link\" data-id=\"https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/logs\/create-custom-table-auxiliary\">Features<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/azure.microsoft.com\/en-us\/pricing\/details\/monitor\/\">Pricing<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Today, we use logging for many purposes including security hunting with SIEM (Sentinel), troubleshooting, performance telemetry, compliance reporting &#8211; but &#8230; <\/p>\n<p class=\"read-more-container\"><a title=\"Optimize Costs using Auxiliary Logs for Verbose Logging\" class=\"read-more button\" href=\"https:\/\/mortenknudsen.net\/?p=3308#more-3308\" aria-label=\"Read more about Optimize Costs using Auxiliary Logs for Verbose Logging\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":3319,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"ngg_post_thumbnail":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[140,55,54,143,141,152,142,134,132,58],"tags":[175,18,104,174,5,9,102,24],"class_list":["post-3308","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ama","category-azure","category-azure-loganalytics","category-azure-logging","category-azure-monitor-agent","category-kusto","category-logging","category-mvpbuzz","category-security","category-sentinel","tag-auxiliary","tag-azure","tag-cost","tag-kusto","tag-loganalytics","tag-logging","tag-optimize","tag-sentinel","infinite-scroll-item","resize-featured-image"],"featured_image_src":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-16.png","author_info":{"display_name":"Morten Knudsen","author_link":"https:\/\/mortenknudsen.net\/?author=1"},"jetpack_featured_media_url":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2024\/09\/image-16.png","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/3308","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3308"}],"version-history":[{"count":18,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/3308\/revisions"}],"predecessor-version":[{"id":3348,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/3308\/revisions\/3348"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/media\/3319"}],"wp:attachment":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3308"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3308"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3308"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}