{"id":1451,"date":"2023-04-02T18:29:59","date_gmt":"2023-04-02T17:29:59","guid":{"rendered":"https:\/\/mortenknudsen.net\/?p=1451"},"modified":"2023-04-03T11:52:29","modified_gmt":"2023-04-03T10:52:29","slug":"009-collecting-iis-logs-using-azure-monitor-agent","status":"publish","type":"post","link":"https:\/\/mortenknudsen.net\/?p=1451","title":{"rendered":"Collecting IIS logs using Azure Monitor Agent"},"content":{"rendered":"\n

This blog will give you insight on how to setup collection of IIS logs<\/strong> from Windows devices<\/strong> using Azure Monitor Agent (AMA).<\/strong><\/p>\n\n\n\n

This blog-post is part of a series of blog posts to master Azure logging in depth (overview)<\/a>. <\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

<\/p>\n\n\n\n

To get you started, you can find ARM-templates & scripts in my AzureLogLibrary (github)<\/a>. Details will be covered in the articles.<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n


<\/p>\n\n\n\n

Quick Links<\/h1>\n\n\n\n

How to start collecting SecurityEvent using ARM-template ?<\/a>
How to setup in GUI<\/a>
High-level architecture \/ flow<\/a>
Tutorial \u2013 How to make data transformations using Data Collection Rules?<\/a>
Data transformation architecture<\/a>
High-level steps to do data transformation<\/a>
Step 1: Kusto command must be defined<\/a>
Escape characters in advanced queries<\/a>
Step 2: Deploy a new DCR<\/a>
Step 3a: Data transformation using AzLogIngestPS<\/a>
Step 3b: Adding the TransformKql using REST API and Powershell (alternative method)<\/a>
Step 4: Verification of changes in DCR (optional)<\/a>
Step 5: Associate the DCR rule to the machine(s)<\/a>
Real-life examples of effect of transformations<\/a>
Example 1 \u2013 removing specific SecurityEvent (5145) from a particular server<\/a>
Example 2 \u2013 removing syslog traffic<\/a><\/p>\n\n\n\n

<\/p>\n\n\n\n

How to start collecting IIS-logs using ARM-template ?<\/h1>\n\n\n\n

You can check out the ARM-templates on my Github<\/a><\/p>\n\n\n\n

IIS-logs – DCE (step 1<\/a>)<\/td><\/tr>
IIS-logs – DCR (step 2)<\/a><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

I do also provide more documentation and powershell script to deploy more DCRs<\/a><\/p>\n\n\n\n

I am also providing ‘Deploy to Azure’ shortcuts, based on the mentioned ARM-templates so you can deploy samples into your environment.<\/p>\n\n\n\n

<\/p>\n\n\n\n

IIS-logs | Deploy DCE to Azure (step 1)<\/h2>\n\n\n\n\n \n<\/a>\n\n\n\n

<\/p>\n\n\n\n

IIS-logs | Deploy DCR-rule to Azure (step 2)<\/h2>\n\n\n\n\n \n<\/a>\n\n\n\n


<\/p>\n\n\n\n

How to setup in GUI ?<\/h1>\n\n\n\n
\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

High-level architecture \/ flow<\/h1>\n\n\n\n

All data will be sent into the Azure LogAnalytics standard table, W3CIISLog<\/strong><\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

The flow is different, as the DCR tells the AMA extension<\/strong> to sent the data to a custom table<\/strong> (Custom-MyTable_CL). Then data is transformed (streamed) into the standard table, W3CIISLog<\/strong>.<\/p>\n\n\n\n

As the data comes into a custom log table initially, we need to have a DCE configured. DCR handles the transformation into the standard table.<\/p>\n\n\n\n

DCR contains the following dataSources<\/strong> section.<\/p>\n\n\n\n

\"dataSources\": {\n            \"iisLogs\": [\n                {\n                    \"streams\": [\n                        \"Microsoft-W3CIISLog\"\n                    ],\n                    \"logDirectories\": [\n                        \"C:\\\\inetpub\\\\logs\\\\LogFiles\\\\W3SVC1\"\n                    ],\n                    \"name\": \"iisLogsDataSource\"\n                }\n            ]<\/code><\/pre>\n\n\n\n
{\n    \"properties\": {\n        \"description\": \"Data collection rule for IIS logs\",\n        \"immutableId\": \"dcr-9631dbaa01214d26b3181772965ae715\",\n        \"dataCollectionEndpointId\": \"\/subscriptions\/xxxxxxxx8-bf1701b862c3\/resourceGroups\/rg-azmon-datacollectionendpoints-p\/providers\/Microsoft.Insights\/dataCollectionEndpoints\/dce-iis-logs-westeurope\",\n        \"streamDeclarations\": {\n            \"Custom-MyTable_CL\": {\n                \"columns\": [\n                    {\n                        \"name\": \"TimeGenerated\",\n                        \"type\": \"datetime\"\n                    },\n                    {\n                        \"name\": \"RawData\",\n                        \"type\": \"string\"\n                    }\n                ]\n            }\n        },\n        \"dataSources\": {\n            \"iisLogs\": [\n                {\n                    \"streams\": [\n                        \"Microsoft-W3CIISLog\"\n                    ],\n                    \"logDirectories\": [\n                        \"C:\\\\inetpub\\\\logs\\\\LogFiles\\\\W3SVC1\"\n                    ],\n                    \"name\": \"iisLogsDataSource\"\n                }\n            ]\n        },\n        \"destinations\": {\n            \"logAnalytics\": [\n                {\n                    \"workspaceResourceId\": \"\/subscriptions\/fce4f282-fcc6-43fb-94d8-bf1701b862c3\/resourcegroups\/rg-logworkspaces\/providers\/microsoft.operationalinsights\/workspaces\/log-platform-management-srvnetworkcloud-p\",\n                    \"workspaceId\": \"b7d80924-d55d-4bf6-b2b3-9889301c7114\",\n                    \"name\": \"DataCollectionIISlogs\"\n                }\n            ]\n        },\n        \"dataFlows\": [\n            {\n                \"streams\": [\n                    \"Microsoft-W3CIISLog\"\n                ],\n                \"destinations\": [\n                    \"DataCollectionIISlogs\"\n                ],\n                \"transformKql\": \"source\",\n                \"outputStream\": \"Microsoft-W3CIISLog\"\n            }\n        ],\n        \"provisioningState\": \"Succeeded\"\n    }<\/code><\/pre>\n\n\n\n

<\/p>\n\n\n\n
<\/p>\n\n\n\n
Tutorial \u2013 How to make data transformations using Data Collection Rules?<\/h1>\n\n\n\n
The below text is embedded from another blog post<\/a><\/p>\n\n\n
\n
This section will show you the steps<\/strong> for setting up data transformations – and how you can do the transformation<\/strong> using AzLogDcrIngestPS<\/a><\/strong> (my new Powershell module) – or using REST API commands in Powershell<\/strong><\/p>\n\n\n\n
If you want to read about transformation in depth, you can find more details here<\/a><\/p>\n\n\n\n
You can also get inspired of real-life samples (and their effects) of using transformation to reduce costs.<\/p>\n\n\n\n
<\/p>\n\n\n\n
Quick Links<\/h1>\n\n\n\n
Data transformation architecture<\/a>
High-level steps to do data transformation<\/a>
Step 1: Kusto command must be defined<\/a>
Escape characters in advanced queries<\/a>
Step 2: Deploy a new DCR<\/a>
Step 3a: Data transformation using AzLogIngestPS<\/a>
Step 3b: Adding the TransformKql using REST API and Powershell (alternative method)<\/a>
Step 4: Verification of changes in DCR (optional)<\/a>
Step 5: Associate the DCR rule to the machine(s)<\/a>
Real-life examples of effect of transformations<\/a>
Example 1 – removing specific SecurityEvent (5145) from a particular server<\/a>
Example 2 – removing syslog traffic<\/a>
<\/p>\n\n\n\n
<\/p>\n\n\n\n
Data transformation architecture<\/h1>\n\n\n\n
Data transformations runs in the Azure Data Ingestion Pipeline<\/strong> and happens very fast as data is being uploaded.<\/p>\n\n\n\n
<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
<\/p>\n\n\n\n
Data transformations are defined in the transformKq<\/strong>l property in the DCR section dataFlows<\/strong>.<\/p>\n\n\n\n
dataFlows\": [\n            {\n                \"streams\": [\n                    \"Microsoft-SecurityEvent\"\n                ],\n                \"destinations\": [\n                    \"DataCollectionEvent\"\n                ],\n                \"transformKql\": \"source | where (EventID != 8002) and (EventID != 5058) and (EventID != 4662) \",<\/strong>\n                \"outputStream\": \"Microsoft-SecurityEvent\"\n            }<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
High-level steps to do data transformation<\/h1>\n\n\n\n
The 5 steps to add a data transformations are:<\/p>\n\n\n\n
    \n
  1. Kusto command must be defined to understand the syntax to exclude the data<\/li>\n\n\n\n
  2. Deploy a new DCR<\/li>\n\n\n\n
  3. Add the transformKql<\/strong> to the DCR – I am covering 2 methods for this in the article (AzLogIngestPS and REST API)<\/li>\n\n\n\n
  4. Verification of changes in DCR (optional)<\/li>\n\n\n\n
  5. Assign the DCR to the machine(s) where the transformation must happen<\/li>\n<\/ol>\n\n\n\n
    <\/p>\n\n\n\n
    In the example below, I am doing a transformation to remove data. Another example is to add new columns, based on incoming data. <\/p>\n\n\n\n
    I hope the below example gives you the insight to understand how to work with transformation.<\/p>\n\n\n\n
    Step 1: Kusto command must be defined<\/h2>\n\n\n\n
    Transformations defines which data should be sent through the pipeline. <\/p>\n\n\n\n
    \n
    If you want to exclude specific events, you will instruct the transformation to exclude these events using standard Kusto – except you will refer to the tablename as source<\/strong> <\/p>\n\n\n\n
    source where (EventID != 4662)<\/strong><\/p>\n<\/blockquote>\n\n\n\n
    <\/p>\n\n\n\n
    Below are 4 samples<\/p>\n\n\n\n
    SecurityEvent | where EventID != 5145<\/td>Here I want to see all Security event except for EventID = 5145<\/td><\/tr>
    SecurityEvent | where (EventID != 8002) and (EventID != 5058) and (EventID != 4662)<\/td>Here I want to see all Security events except for EventID = 8002,5058,4662<\/td><\/tr>
    Event | where ( (EventID != 10016 and EventLog == \u201cApplication\u201d)  )<\/td>Here I want to see all Event system and application events, except for application events with eventid 10016<\/td><\/tr>
    CommonSecurityLog | where (DeviceVendor !contains \u201csonicwall\u201d) or ((DeviceVendor contains \u201csonicwall\u201d) and (Activity contains \u201cconnection opened\u201d or Activity contains \u201cconnection closed\u201d) and (Protocol != \u201cudp\/dns\u201d))<\/td>Here I want to see all CEF\/syslog where devicevendor is different from sonicwall like Cisco and all sonicwall events, except if protocol is udp\/dns<\/td><\/tr><\/tbody><\/table>

    <\/figcaption><\/figure>\n\n\n\n
    \n
    Since the transformation is applied to each record individually, it can\u2019t use any KQL operators that act on multiple records. Only operators that take a single row as input and return no more than one row are supported. <\/p>\n\n\n\n
    For example, summarize<\/a> isn\u2019t supported since it summarizes multiple records. See Supported KQL features<\/a> for a complete list of supported features.<\/p>\n\n\n\n
    Therefore you cannot use cross-workspace references like doing lookup in another table.<\/p>\n<\/blockquote>\n\n\n\n
    <\/p>\n\n\n\n
    When the Kusto command is working as expected, then change the tablename to source<\/strong>.<\/p>\n\n\n\n
    SecurityEvent | where EventID != 5145<\/code><\/pre>\n\n\n\n
    was changed to<\/p>\n\n\n\n
    source | where EventID != 5145<\/code><\/pre>\n\n\n\n
    <\/p>\n\n\n\n
    Escape characters in advanced queries<\/h2>\n\n\n\n
    If you need to work with advanced queries, it can be required to adjust the query with escape characters:<\/p>\n\n\n\n
    SecurityEvent | where (Account != \u201cWINSFTP\\\\autotest\u201d) and (EventID != 4688 and EventID != 8002 and EventID != 4625) and (Account != \u201cWORKGROUP\\\\WINSFTP$\u201d)<\/code><\/pre>\n\n\n\n
    will be changed to<\/p>\n\n\n\n
    source\\r\\n| where (Account != \\\u201dCVT-WINSFTP\\\\\\\\cvtautotest\\\u201d) and (EventID != 4688 and EventID != 8002 and EventID != 4625) and (Account != \\\u201dWORKGROUP\\\\\\\\CVT-WINSFTP$\\\u201d)<\/code><\/pre>\n\n\n\n
    Using online converter<\/strong><\/h3>\n\n\n\n
    In case you are working with advanced queries, I prefer to take my Kusto query and paste it in an online converter, which will convert the query with escape characters.<\/p>\n\n\n\n
    Personally, I use the online website https:\/\/jsonformatter.org\/json-escape<\/a><\/p>\n\n\n\n
    <\/p>\n\n\n\n
    Step 2: Deploy a new DCR<\/h2>\n\n\n\n
    Configure a standard DCR rule and choose to collect what you want like for example all security events or all system and application events.<\/p>\n\n\n\n
    Make a note of the ResourceId<\/strong> of the DCR rule. You will use the ResourceId in step 3<\/p>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    \"\"<\/figure>\n\n\n\n

    <\/p>\n\n\n\n
    Step 3a: Data transformation using AzLogIngestPS<\/h2>\n\n\n\n
    It is very easy to work with transformations using my powershell module, AzLogIngestPS<\/strong>. <\/p>\n\n\n\n
    You can read about my powershell module, AzLogIngestPS here<\/a><\/p>\n\n\n\n
    To get started viewing an existing transformation, you will only require the ResourceId, which was noted in the previous step 2.<\/p>\n\n\n\n
    You can view an existing data transformations (transformKql) using the function Get-AzDataCollectionRuleTransformKql<\/strong><\/p>\n\n\n\n
    Get-AzDataCollectionRuleTransformKql  -DcrResourceID <DCR resourceid><\/code><\/pre>\n\n\n\n
    Here you can see value of the transformKql marked in bold (source ….)<\/p>\n\n\n\n
    PS > Get-AzDataCollectionRuleTransformKql -DcrResourceId \/subscriptions\/fxxxxx4d8-bf1701b862c3\/resourceGroups\/rg-logworkspaces\/providers\/microsoft.insights\/dataCollectionRules\/dcr-ingest-exclude-security-eventid\nsource | where (EventID != 8002) and (EventID != 5058) and (EventID != 4662) and (EventID != 4688)<\/strong><\/code><\/pre>\n\n\n\n
    You can add a transformation to an existing DCR using the function Update-AzDataCollectionRuleTransformKql<\/strong><\/p>\n\n\n\n
    Update-AzDataCollectionRuleTransformKql -DcrResourceId <DCR-ID> -transformKql  <transform-KQL><\/code><\/pre>\n\n\n\n
    Here you can see the value before <\/strong>and after<\/strong> updating the transformation.<\/p>\n\n\n\n
    Value before<\/p>\n\n\n\n
    PS > Get-AzDataCollectionRuleTransformKql -DcrResourceId \/subscriptions\/fce4f282-fcc6-43fb-94d8-bf1701b862c3\/resourceGroups\/rg-logworkspaces\/providers\/microsoft.insights\/dataCollectionRules\/dcr-ingest-exclude-security-eventid\nsource | where (EventID != 8002) and (EventID != 5058) and (EventID != 4662) and (EventID != 4688)<\/code><\/pre>\n\n\n\n
    Adding the transformation<\/p>\n\n\n\n
    PS > Update-AzDataCollectionRuleTransformKql -DcrResourceId \/subscriptions\/xxxxxx-43fb-94d8-bf1701b862c3\/resourceGroups\/rg-logworkspaces\/providers\/microsoft.insights\/dataCollectionRules\/dcr-ingest-exclude-security-eventid -transformKql  \"source | where (EventID != 8002) and (EventID != 5058) and (EventID != 4662) and (EventID != 4688) and (EventID != 4663)\"<\/strong>\n\nUpdating transformKql for DCR\n\/subscriptions\/xxxxxfb-94d8-bf1701b862c3\/resourceGroups\/rg-logworkspaces\/providers\/microsoft.insights\/dataCollectionRules\/dcr-ingest-exclude-security-eventid<\/code><\/pre>\n\n\n\n
    Value after – note “and (EventID != 4663)<\/strong>” has been added compared to before value<\/p>\n\n\n\n
    PS > Get-AzDataCollectionRuleTransformKql -DcrResourceId \/subscriptions\/xxxxx3fb-94d8-bf1701b862c3\/resourceGroups\/rg-logworkspaces\/providers\/microsoft.insights\/dataCollectionRules\/dcr-ingest-exclude-security-eventid\nsource | where (EventID != 8002) and (EventID != 5058) and (EventID != 4662) and (EventID != 4688) and (EventID != 4663)<\/strong><\/code><\/pre>\n\n\n\n
    You can install AzLogDcrIngestPS<\/strong> from Powershell gallery<\/strong><\/p>\n\n\n\n
    install-module AzLogIngestPS<\/code><\/pre>\n\n\n\n
    You need to connect to Azure first with an account with RBAC permissions.<\/p>\n\n\n\n
    PS > Connect-AzAccount<\/code><\/pre>\n\n\n\n

    <\/p>\n\n\n\n
    <\/p>\n\n\n\n
    Step 3b: Adding the TransformKql using REST API and Powershell (alternative method)<\/h2>\n\n\n\n
    As an alternative to using AzLogIngestPS, you can also call REST API and add the transformKql<\/strong> command.<\/p>\n\n\n\n
    TransformKql requires a REST API call to minimum api-version 2021-09-01-preview<\/strong>. You can see the syntax below.<\/p>\n\n\n\n
    Currently, the recommended API is 2022-06-01<\/strong>. <\/p>\n\n\n\n
    You can see available API version in the GUI.<\/p>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    <\/p>\n\n\n\n
    \"\"<\/figure>\n\n\n\n

    <\/p>\n\n\n\n
    The steps to add a transformation to an existing DCR are:<\/p>\n\n\n\n
      \n
    1. Retrieve the entire DCR using REST API (GET) in JSON format – and save it to a TXT file<\/li>\n\n\n\n
    2. Edit the file – and add the transformKql<\/strong> parameter in the dataFlow<\/strong> section<\/li>\n\n\n\n
    3. Upload the entire file content using REST API (PUT)<\/li>\n<\/ol>\n\n\n\n
      <\/p>\n\n\n\n
      I have provided a Powershell script on my github<\/a> to retrieve the DCR into a flat JSON-file so you can edit the file and add the needed transformation \u2013 and then upload the modified DCR again. <\/p>\n\n\n\n
      Below I have created a sample folder C:\\TMP where the file will be stored<\/p>\n\n\n\n
      <\/p>\n\n\n\n
      Start by setting the variables $ResourceId and $FilePath<\/strong><\/p>\n\n\n\n
      ResourceId was noted in the previous step 2<\/p>\n\n\n\n
      FilePath can be any path \u2013 it is only a temporary file used for this change as example<\/p>\n\n\n\n
      ####################################################\n# VARIABLES\n####################################################\n\n# here you put the ResourceID of the Data Collection Rules (a sample is provided below)\n$ResourceId = \"\/subscriptions\/xxxxxx\/resourceGroups\/rg-logworkspaces\/providers\/microsoft.insights\/dataCollectionRules\/dcr-ingest-exclude-security-eventid\"\n    \n# here you put a path and file name where you want to store the temporary file-extract from DCR (a sample is provided below)\n$FilePath   = \"c:\\tmp\\dcr-ingest-exclude-security-eventid.txt\"\n<\/code><\/pre>\n\n\n\n
      Connect to Azure<\/strong><\/p>\n\n\n\n
      ####################################################\n# CONNECT TO AZURE\n####################################################\n\nConnect-AzAccount<\/code><\/pre>\n\n\n\n
      Run the export DCR<\/strong><\/p>\n\n\n\n
      $DCR = Invoke-AzRestMethod -Path (\"$ResourceId\"+\"?api-version=2022-06-01\") -Method GET\n\n$DCR.Content | ConvertFrom-Json | ConvertTo-Json -Depth 20 | Out-File -FilePath $FilePath\n<\/code><\/pre>\n\n\n\n
      Now you have a JSON file in c:\\tmp folder<\/p>\n\n\n\n
      Modify file and add TransformKql<\/strong><\/p>\n\n\n\n
      Open the file using your favorite editor and add the line transformKql<\/strong> command that you created in step 1<\/p>\n\n\n\n
      \"transformKql\":  \"source\\n| where (EventID != 5145)\"<\/strong>,<\/code><\/pre>\n\n\n\n
      NOTE: Remember to add the , (comma) at the end of the line so you are not breaking the JSON syntax.<\/p>\n\n\n\n
      \"dataFlows\": [\n            {\n                \"streams\": [\n                    \"Microsoft-SecurityEvent\"\n                ],\n                \"destinations\": [\n                    \"DataCollectionEvent\"\n                ],\n                \"transformKql\": \"source | where (EventID != 8002) and (EventID != 4688) and (EventID != 4663)\",<\/strong>\n                \"outputStream\": \"Microsoft-SecurityEvent\"\n            }\n        ],<\/code><\/pre>\n\n\n\n
      <\/p>\n\n\n\n
      Upload the modified DCR (overwrite)<\/strong><\/p>\n\n\n\n
      Now you want to run the last part of the powershell script, which will update the DCR taking the entire content of the local file and making PUT REST call against the specific api version 2022-06-01<\/strong>. <\/p>\n\n\n\n
      <\/p>\n\n\n\n
      ####################################################\n# UPLOAD FILE \/ UPDATE DCR WITH TRANSFORM\n####################################################\n\n$DCRContent = Get-Content $FilePath -Raw \n\nInvoke-AzRestMethod -Path (\"$ResourceId\"+\"?api-version=2022-06-01\") -Method PUT -Payload $DCRContent\n<\/code><\/pre>\n\n\n\n
      You should be getting a StatusCode 200 with the PUT commmand, indicating everything it updated correctly<\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      <\/p>\n\n\n\n
      If there is is an error in the file structure, you will get an error 400<\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n

      <\/p>\n\n\n\n
      Step 4: Verification of changes in DCR (optional)<\/h2>\n\n\n\n
      You can choose to check in the GUI for the change. Remember to choose the API-version 2022-06-01 (or 2021-09-01-preview). Otherwise you wont be able to see the change. <\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      <\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      <\/p>\n\n\n\n
      You will now see the transformKql.<\/p>\n\n\n\n
      You can also extract the changes running the first lines again to extract into the local file using the GET command<\/p>\n\n\n\n
      ####################################################\n# EXPORT EXISTING DCR TO FILE\n####################################################\n\n$DCR = Invoke-AzRestMethod -Path (\"$ResourceId\"+\"?api-version=2022-06-01\") -Method GET\n\n$DCR.Content | ConvertFrom-Json | ConvertTo-Json -Depth 20 | Out-File -FilePath $FilePath\n<\/code><\/pre>\n\n\n\n

      <\/p>\n\n\n\n
      Step 5: Associate the DCR rule to the machine(s)<\/h2>\n\n\n\n
      Lastly you have to associate the DCR rule to the machine(s) where you want the transformation to happen.<\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      <\/p>\n\n\n\n
      You have to wait for pipeline transformation to happen. Normally it will start within 5 minutes, sometimes a little longer.<\/p>\n\n\n\n

      <\/p>\n\n\n\n
      Real-life examples of effect of transformations<\/h1>\n\n\n\n
      Below you can find some examples and their effect of using transformations – with focus on cost reductions. <\/p>\n\n\n\n
      As you can see, the results were a significant decrease.<\/p>\n\n\n\n
      Of course there is a trade-off, which must be considered. There are always 2 sides of the coin \u2013 will I miss the data at some point !<\/p>\n\n\n\n
      <\/p>\n\n\n\n
      Example 1 – removing specific SecurityEvent (5145) from a particular server<\/h2>\n\n\n\n
      <\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      <\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      The effect in cost – here shown in DKK, where the daily cost drops to DKK 2300 from DKK 5000<\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      <\/p>\n\n\n\n
      Example 2 – removing syslog traffic<\/h2>\n\n\n\n
      Here is an example where I did a transformation removing Syslog events<\/strong> with specific patterns.<\/p>\n\n\n\n
      source\n| where (DeviceVendor !contains \"sonicwall\") or ((DeviceVendor contains \"sonicwall\") and (Activity contains \"connection opened\" or Activity contains \"connection closed\") and (Protocol != \"udp\/dns\"))<\/code><\/pre>\n\n\n\n
      <\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      <\/p>\n<\/div>","protected":false},"excerpt":{"rendered":"
      This blog will give you insight on how to setup collection of IIS logs from Windows devices using Azure Monitor … <\/p>\n
      Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":1899,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"ngg_post_thumbnail":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[140,55,59,129,131,61,128,54,143,141,57,130,28,139,136,138,137,142,133,134,146,60,132,58,145,135,144],"tags":[],"class_list":["post-1451","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ama","category-azure","category-azure-arc","category-azure-data-collection-rules","category-azure-data-ingestion-pipeline","category-azure-extensions","category-azure-log-ingestion-api","category-azure-loganalytics","category-azure-logging","category-azure-monitor-agent","category-azure-security","category-clientinspector","category-community","category-data-collection-endpoint","category-data-transformation","category-dce","category-dcr","category-logging","category-microsoft-security","category-mvpbuzz","category-performance-collection","category-scripting","category-security","category-sentinel","category-servicemap","category-transformation","category-vminsight","infinite-scroll-item","resize-featured-image"],"featured_image_src":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-11.png","author_info":{"display_name":"Morten Knudsen","author_link":"https:\/\/mortenknudsen.net\/?author=1"},"jetpack_featured_media_url":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-11.png","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1451","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1451"}],"version-history":[{"count":17,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1451\/revisions"}],"predecessor-version":[{"id":2145,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1451\/revisions\/2145"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/media\/1899"}],"wp:attachment":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1451"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1451"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1451"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}