{"id":1171,"date":"2023-04-07T11:07:59","date_gmt":"2023-04-07T10:07:59","guid":{"rendered":"https:\/\/mortenknudsen.net\/?p=1171"},"modified":"2023-04-10T11:38:19","modified_gmt":"2023-04-10T10:38:19","slug":"azlogdcringestps-fixing-invalid-datasources-data-manipulation-before-sending-data-via-azure-pipeline-log-ingestion-api-azure-data-collection-rules-into-azure-loganalytics","status":"publish","type":"post","link":"https:\/\/mortenknudsen.net\/?p=1171","title":{"rendered":"AzLogDcrIngestPS &#8211; how to do data manipulation before sending data via Azure Pipeline, Log ingestion API &#038; Azure Data Collection Rules into Azure LogAnalytics ?"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">If you are sending data using <strong>HTTP Data Collector API<\/strong> <strong>(REST)<\/strong> today, you should continue reading, as this API will be deprecated, as part of the transition to <strong>Log ingestion API<\/strong> using <strong>Azure Data Collection Rules<\/strong>, <strong>Azure Pipeline<\/strong>, <strong>Azure LogAnalytics custom tables (v2)<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"592\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-35-1024x592.png\" alt=\"\" class=\"wp-image-2250\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-35-1024x592.png 1024w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-35-300x174.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-35-768x444.png 768w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-35.png 1058w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">As you can see from the illustration more components (DCR, DCR, Pipeline, Schema) are added, which also <strong>increases the complexity<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I have built a Powershell module, <a href=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS\" target=\"_blank\" rel=\"noreferrer noopener\">AzLogDcrIngestPS<\/a> which will ease the steps, if you want to send any data to&nbsp;<strong>Azure LogAnalytics custom logs (v2)<\/strong>&nbsp;&#8211; using the new features of&nbsp;<strong>Azure Log Ingestion Pipeline<\/strong>,&nbsp;<strong>Azure Data Colection Rules &amp; Log Ingestion API<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>AzLogDcrIngestPS<\/strong> includes <strong>25 functions<\/strong> dealing with:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>data manipulation<\/strong>&nbsp;before sending data in (7 functions)<\/li>\n\n\n\n<li><strong>table \/ dcr \/ schema \/ transformation management<\/strong>&nbsp;(13 functions)<\/li>\n\n\n\n<li><strong>data upload<\/strong>&nbsp;using Azure Log Ingestion Pipeline \/ Log Ingestion API (4 functions)<\/li>\n\n\n\n<li><strong>support\/security<\/strong>&nbsp;(1 function)<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This blog post will go in more details about how you can <strong>manipulate your data<\/strong> using functions in <strong><a href=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS\" target=\"_blank\" rel=\"noreferrer noopener\">AzLogDcrIngestPS<\/a><\/strong> so data can be uploaded into <strong>Azure LogAnalytics<\/strong> using <strong>Log Ingestion API<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you want to get started or want more information, <a href=\"https:\/\/mortenknudsen.net\/?p=1148\" target=\"_blank\" rel=\"noreferrer noopener\">check out this blog-post<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Understanding your data<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Before sending data into Azure LogAnalytics, it is obvious that you want to ensure your data is containing <strong>valuable data<\/strong> and <strong>structured<\/strong> so you can access them afterwards.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A good naming convention is also important, as data must be uploaded based upon a schema, both for DCR and LogAnalytics. LogAnalytics and DCR have prohibited properties (column-names in table).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In case data is coming from sources like CIM, WMI, etc. they will contain information, where I typically strip the extra data as it is just &#8220;noice&#8221;.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS\/blob\/main\/docs\/Architecture-simpleoverview.png\" target=\"_blank\" rel=\"noreferrer noopener\"><img decoding=\"async\" src=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS\/raw\/main\/docs\/Architecture-simpleoverview.png\" alt=\"Overview\"\/><\/a><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you want to see the functions in action, check out the video below<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/youtu.be\/OZWj7xZHLI8\">Video 2m 19s &#8211; Data manipulation<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">See AzLogDcrIngestPS functions in action \ud83d\ude42<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I have&nbsp;<a href=\"https:\/\/mortenknudsen.net\/?p=1148#getstarted\">provided 4 demos<\/a>&nbsp;for you to try. I have also provided&nbsp;<a href=\"https:\/\/mortenknudsen.net\/?p=1148#videos\">videos<\/a>&nbsp;for you to check out.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Alternatively, I have built a cool showcase \u2013&nbsp;<a href=\"https:\/\/github.com\/KnudsenMorten\/ClientInspectorV2\">ClientInspector (v2)<\/a>,&nbsp;<strong>free for the community<\/strong>, where you can see how you can use the funtions from the&nbsp;<strong>AzLogDcrIngestPS<\/strong>&nbsp;module.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS\/blob\/main\/img\/ClientInspector_300.png\" target=\"_blank\" rel=\"noreferrer noopener\"><img decoding=\"async\" src=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS\/raw\/main\/img\/ClientInspector_300.png\" alt=\"ClientInspector\"\/><\/a><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/github.com\/KnudsenMorten\/ClientInspectorV2\">ClientInspector<\/a>&nbsp;can bring back data from your clients using&nbsp;<strong>Azure Log Ingestion Pipeline<\/strong>,&nbsp;<strong>Azure Data Collection Rules<\/strong>,&nbsp;<strong>Azure LogAnalytics<\/strong>; view them with&nbsp;<strong>Azure Monitor &amp; Azure Dashboards<\/strong>&nbsp;\u2013 and get \u201cdrift-alerts\u201d using&nbsp;<strong>Microsoft Sentinel<\/strong>. it includes tons of great information and dashboards to see if you are in control with your clients \u2013 or something is drifting from desired state.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h1 class=\"wp-block-heading\">Data manipulation functions<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">AzLogDcrIngestPS module provides 7 functions shown below.<\/p>\n\n\n\n<figure class=\"wp-block-table has-small-font-size\"><table class=\"has-fixed-layout\"><thead><tr><th>Function name<\/th><th>Synopsis<\/th><\/tr><\/thead><tbody><tr><td><a href=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS#add-collectiontimetoallentriesinarray\" target=\"_blank\" rel=\"noreferrer noopener\">Add-CollectionTimeToAllEntriesInArray<\/a><\/td><td>Add property CollectionTime (based on current time) to all entries on the object<\/td><\/tr><tr><td><a href=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS#add-columndatatoallentriesinarray\" target=\"_blank\" rel=\"noreferrer noopener\">Add-ColumnDataToAllEntriesInArray<\/a><\/td><td>Adds up to 3 extra columns and data to the object<\/td><\/tr><tr><td><a href=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS#build-dataarraytoalignwithschema\" target=\"_blank\" rel=\"noreferrer noopener\">Build-DataArrayToAlignWithSchema<\/a><\/td><td>Rebuilds the source object to match modified schema structure &#8211; used after usage of ValidateFix-AzLogAnalyticsTableSchemaColumnNames<\/td><\/tr><tr><td><a href=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS#convert-cimarraytoobjectfixstructure\" target=\"_blank\" rel=\"noreferrer noopener\">Convert-CimArrayToObjectFixStructure<\/a><\/td><td>Converts CIM array and remove CIM class information<\/td><\/tr><tr><td><a href=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS#convert-psarraytoobjectfixstructure\" target=\"_blank\" rel=\"noreferrer noopener\">Convert-PSArrayToObjectFixStructure<\/a><\/td><td>Converts PS array and remove PS class information<\/td><\/tr><tr><td><a href=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS#filter-objectexcludeproperty\" target=\"_blank\" rel=\"noreferrer noopener\">Filter-ObjectExcludeProperty<\/a><\/td><td>Removes columns from the object which is considered &#8220;noice&#8221; and shouldn&#8217;t be send to logs<\/td><\/tr><tr><td><a href=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS#validatefix-azloganalyticstableschemacolumnnames\" target=\"_blank\" rel=\"noreferrer noopener\">ValidateFix-AzLogAnalyticsTableSchemaColumnNames<\/a><\/td><td>Validates the column names in the schema are valid according the requirement for LogAnalytics tables.<br>Fixes any issues by rebuild the source object<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h1 class=\"wp-block-heading\">How can I get access to the help, parameters, syntax, examples &#8211; using get-help ?<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Get help with a specific cmdlet with the command<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>get-help Add-CollectionTimeToAllEntriesInArray -full<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code>PS&gt; get-help Add-CollectionTimeToAllEntriesInArray -full\n\nNAME\n    Add-CollectionTimeToAllEntriesInArray\n    \nSYNOPSIS\n    Add property CollectionTime (based on current time) to all entries on the object\n    \n    \nSYNTAX\n    Add-CollectionTimeToAllEntriesInArray &#91;-Data] &lt;Array&gt; &#91;&lt;CommonParameters&gt;]\n    \n    \nDESCRIPTION\n    Gives capability to do proper searching in queries to find latest set of records with same collection time\n    Time Generated cannot be used when you are sending data in batches, as TimeGenerated will change\n    An example where this is important is a complete list of applications for a computer. We want all applications to\n    show up when queriying for the latest data\n    \n\nPARAMETERS\n    -Data &lt;Array&gt;\n        Object to modify\n        \n        Required?                    true\n        Position?                    1\n        Default value                \n        Accept pipeline input?       false\n        Accept wildcard characters?  false\n        \n    &lt;CommonParameters&gt;\n        This cmdlet supports the common parameters: Verbose, Debug,\n        ErrorAction, ErrorVariable, WarningAction, WarningVariable,\n        OutBuffer, PipelineVariable, and OutVariable. For more information, see \n        about_CommonParameters (https:\/go.microsoft.com\/fwlink\/?LinkID=113216). \n    \nINPUTS\n    None. You cannot pipe objects\n    \n    \nOUTPUTS\n    Updated object with CollectionTime\n    \n    \n    -------------------------- EXAMPLE 1 --------------------------\n    \n    PS C:\\&gt;#-------------------------------------------------------------------------------------------\n    \n    # Variables\n    #-------------------------------------------------------------------------------------------\n    $Verbose                   = $true  # $true or $false\n    \n    #-------------------------------------------------------------------------------------------\n    # Collecting data (in)\n    #-------------------------------------------------------------------------------------------\n    $DNSName                   = (Get-CimInstance win32_computersystem).DNSHostName +\".\" + (Get-CimInstance win32_computersystem).Domain\n    $ComputerName              = (Get-CimInstance win32_computersystem).DNSHostName\n    &#91;datetime]$CollectionTime  = ( Get-date (&#91;datetime]::Now.ToUniversalTime()) -format \"yyyy-MM-ddTHH:mm:ssK\" )\n    \n    $UserLoggedOnRaw           = Get-Process -IncludeUserName -Name explorer | Select-Object UserName -Unique\n    $UserLoggedOn              = $UserLoggedOnRaw.UserName\n    \n    $DataVariable = Get-CimInstance -ClassName Win32_Processor | Select-Object -ExcludeProperty \"CIM*\"\n    \n    #-------------------------------------------------------------------------------------------\n    # Preparing data structure\n    #-------------------------------------------------------------------------------------------\n    $DataVariable = Convert-CimArrayToObjectFixStructure -data $DataVariable -Verbose:$Verbose\n    $DataVariable\n    \n    # add CollectionTime to existing array\n    $DataVariable = Add-CollectionTimeToAllEntriesInArray -Data $DataVariable -Verbose:$Verbose\n    $DataVariable\n    \n    #-------------------------------------------------------------------------------------------\n    # Output\n    #-------------------------------------------------------------------------------------------\n    \n    VERBOSE:   Adding CollectionTime to all entries in array .... please wait !\n    Caption                                 : Intel64 Family 6 Model 165 Stepping 5\n    Description                             : Intel64 Family 6 Model 165 Stepping 5\n    InstallDate                             : \n    Name                                    : Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz\n    Status                                  : OK\n    Availability                            : 3\n    ConfigManagerErrorCode                  : \n    ConfigManagerUserConfig                 : \n    CreationClassName                       : Win32_Processor\n    DeviceID                                : CPU0\n    ErrorCleared                            : \n    ErrorDescription                        : \n    LastErrorCode                           : \n    PNPDeviceID                             : \n    PowerManagementCapabilities             : \n    PowerManagementSupported                : False\n    StatusInfo                              : 3\n    SystemCreationClassName                 : Win32_ComputerSystem\n    SystemName                              : STRV-MOK-DT-02\n    AddressWidth                            : 64\n    CurrentClockSpeed                       : 2904\n    DataWidth                               : 64\n    Family                                  : 198\n    LoadPercentage                          : 1\n    MaxClockSpeed                           : 2904\n    OtherFamilyDescription                  : \n    Role                                    : CPU\n    Stepping                                : \n    UniqueId                                : \n    UpgradeMethod                           : 1\n    Architecture                            : 9\n    AssetTag                                : To Be Filled By O.E.M.\n    Characteristics                         : 252\n    CpuStatus                               : 1\n    CurrentVoltage                          : 8\n    ExtClock                                : 100\n    L2CacheSize                             : 2048\n    L2CacheSpeed                            : \n    L3CacheSize                             : 16384\n    L3CacheSpeed                            : 0\n    Level                                   : 6\n    Manufacturer                            : GenuineIntel\n    NumberOfCores                           : 8\n    NumberOfEnabledCore                     : 8\n    NumberOfLogicalProcessors               : 16\n    PartNumber                              : To Be Filled By O.E.M.\n    ProcessorId                             : BFEBFBFF000A0655\n    ProcessorType                           : 3\n    Revision                                : \n    SecondLevelAddressTranslationExtensions : False\n    SerialNumber                            : To Be Filled By O.E.M.\n    SocketDesignation                       : U3E1\n    ThreadCount                             : 16\n    Version                                 : \n    VirtualizationFirmwareEnabled           : False\n    VMMonitorModeExtensions                 : False\n    VoltageCaps                             : \n    PSComputerName                          : \n    CollectionTime                          : 12-03-2023 16:08:33\n    \n    \n    \n    \n    \nRELATED LINKS\n    https:&#47;&#47;github.com\/KnudsenMorten\/AzLogDcrIngestPS<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h1 class=\"wp-block-heading\">Download latest version<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">You can download latest version of AzLogDcrIngestPS here \u2013 or install from Powershell Gallery:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/www.powershellgallery.com\/packages\/AzLogDcrIngestPS\" target=\"_blank\" rel=\"noreferrer noopener\">Install AzLogDcringestPS from Powershell Gallery<\/a><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>install-module AzLogDcrIngestPS<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/www.powershellgallery.com\/packages\/AzLogDcrIngestPS\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"998\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-33-1024x998.png\" alt=\"\" class=\"wp-image-2225\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-33-1024x998.png 1024w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-33-300x292.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-33-768x749.png 768w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-33.png 1236w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/raw.githubusercontent.com\/KnudsenMorten\/AzLogDcrIngestPS\/main\/AzLogDcrIngestPS.psm1\" target=\"_blank\" rel=\"noreferrer noopener\">Download AzLogDcringestPS module from this Github repositry<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>If you are sending data using HTTP Data Collector API (REST) today, you should continue reading, as this API will &#8230; <\/p>\n<p class=\"read-more-container\"><a title=\"AzLogDcrIngestPS &#8211; how to do data manipulation before sending data via Azure Pipeline, Log ingestion API &#038; Azure Data Collection Rules into Azure LogAnalytics ?\" class=\"read-more button\" href=\"https:\/\/mortenknudsen.net\/?p=1171#more-1171\" aria-label=\"Read more about AzLogDcrIngestPS &#8211; how to do data manipulation before sending data via Azure Pipeline, Log ingestion API &#038; Azure Data Collection Rules into Azure LogAnalytics ?\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":2192,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"ngg_post_thumbnail":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[129,131,128,54,143,57,28,137,142,133,134,60,132,58,1],"tags":[],"class_list":["post-1171","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure-data-collection-rules","category-azure-data-ingestion-pipeline","category-azure-log-ingestion-api","category-azure-loganalytics","category-azure-logging","category-azure-security","category-community","category-dcr","category-logging","category-microsoft-security","category-mvpbuzz","category-scripting","category-security","category-sentinel","category-uncategorized","infinite-scroll-item","resize-featured-image"],"featured_image_src":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/datamanipulation-1.png","author_info":{"display_name":"Morten Knudsen","author_link":"https:\/\/mortenknudsen.net\/?author=1"},"jetpack_featured_media_url":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/datamanipulation-1.png","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1171","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1171"}],"version-history":[{"count":9,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1171\/revisions"}],"predecessor-version":[{"id":2259,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1171\/revisions\/2259"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/media\/2192"}],"wp:attachment":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1171"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1171"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1171"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}