{"id":1499,"date":"2023-04-07T12:16:24","date_gmt":"2023-04-07T11:16:24","guid":{"rendered":"https:\/\/mortenknudsen.net\/?p=1499"},"modified":"2023-04-10T15:33:05","modified_gmt":"2023-04-10T14:33:05","slug":"azlogdcringestps-tips-tricks-sending-data-via-azure-pipeline-azure-log-ingestion-api-azure-data-collections-into-azure-loganalytics","status":"publish","type":"post","link":"https:\/\/mortenknudsen.net\/?p=1499","title":{"rendered":"AzLogDcrIngestPS &#8211; tips &#038; tricks sending data via Azure Pipeline, Azure Log Ingestion API, Azure Data Collections into Azure LogAnalytics"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">If you are sending data using <strong>HTTP Data Collector API<\/strong> <strong>(REST)<\/strong> today, you should continue reading, as this API will be deprecated, as part of the transition to <strong>Log ingestion API<\/strong> using <strong>Azure Data Collection Rules<\/strong>, <strong>Azure Pipeline<\/strong>, <strong>Azure LogAnalytics custom tables (v2)<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"592\" src=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-35-1024x592.png\" alt=\"\" class=\"wp-image-2250\" srcset=\"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-35-1024x592.png 1024w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-35-300x174.png 300w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-35-768x444.png 768w, https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/image-35.png 1058w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As you can see from the illustration more components (DCR, DCR, Pipeline, Schema) are added, which also <strong>increases the complexity<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I have built a Powershell module, <a href=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS\" target=\"_blank\" rel=\"noreferrer noopener\">AzLogDcrIngestPS<\/a> which will ease the steps, if you want to send any data to&nbsp;<strong>Azure LogAnalytics custom logs (v2)<\/strong>&nbsp;&#8211; using the new features of&nbsp;<strong>Azure Log Ingestion Pipeline<\/strong>,&nbsp;<strong>Azure Data Colection Rules &amp; Log Ingestion API<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>AzLogDcrIngestPS<\/strong> includes <strong>25 functions<\/strong> dealing with:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>data manipulation<\/strong>&nbsp;before sending data in (7 functions)<\/li>\n\n\n\n<li><strong>table \/ dcr \/ schema \/ transformation management<\/strong>&nbsp;(13 functions)<\/li>\n\n\n\n<li><strong>data upload<\/strong>&nbsp;using Azure Log Ingestion Pipeline \/ Log Ingestion API (4 functions)<\/li>\n\n\n\n<li><strong>support\/security<\/strong>&nbsp;(1 function)<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">This blog post will cover <strong>tips &amp; troubleshooting options<\/strong> using functions in&nbsp;<strong><a href=\"https:\/\/github.com\/KnudsenMorten\/AzLogDcrIngestPS\" target=\"_blank\" rel=\"noreferrer noopener\">AzLogDcrIngestPS<\/a><\/strong>&nbsp;so data can be uploaded into&nbsp;<strong>Azure LogAnalytics<\/strong>&nbsp;using&nbsp;<strong>Log Ingestion API<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Error: The location property is required for this definition<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">If you receive an error similar to the below error, this is caused by the script cannot find the DceName used. Check for spelling mistakes<\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code>invoke-webrequest : {\"error\":{\"code\":\"LocationRequired\",\"message\":\"The location property is required for this definition.\"}}\r\nAt C:\\Program Files\\WindowsPowerShell\\Modules\\AzLogDcrIngestPS\\1.2.37\\AzLogDcrIngestPS.psm1:1635 char:21\r\n+ ...             invoke-webrequest -UseBasicParsing -Uri $Uri -Method PUT  ...\r\n+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) &#91;Invoke-WebRequest], WebException\r\n    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand\r\n\r<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h1 class=\"wp-block-heading\">Error 513 &#8211; entity is too large &#8211; entity is too large<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">By default the Post functions will send the data in batches depending on an calculated average size per record. In case your recordset is of different size, you might receive an error 513.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Cause is that you are hitting the limitation of 1 mb for each upload (Azure Pipeline limitation). Microsoft wants to receive many smaller chunks of data, as this is a shared environment. I have seen this issue when retrieving the list of all installed applications. Apparently the applications are storing information of very different degree of size.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can mitigate this issue, by adding the parameter&nbsp;<strong>-BatchAmount&nbsp;<\/strong>to the Post-command. If you want to be sure, set it to 1<\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code>Post-AzLogAnalyticsLogIngestCustomLogDcrDce-Output -DceName $DceName `\n                                                   -DcrName $DcrName `\n                                                   -Data $DataVariable `\n                                                   -TableName $TableName `\n                                                   -AzAppId $LogIngestAppId `\n                                                   -AzAppSecret $LogIngestAppSecret `\n                                                   -TenantId $TenantId `\n<strong>                                                   -BatchAmount 1 `<\/strong>\n\t\t\t\t\t\t   -Verbose:$Verbose<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h1 class=\"wp-block-heading\">Verbose-mode<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">If you want to get more detailed information about that is happening, you can enable verbose mode (-verbose:$true)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here is an example with ClientInspector, where verbose mode is activated.<\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code>.\\ClientInspector.ps1 -verbose:$true -function:localpath<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Here is an example where the function is called in verbose-mode<\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code>Build-DataArrayToAlignWithSchema -Data $DataVariable -Verbose:$true<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h1 class=\"wp-block-heading\">How can I see which cmdlets are available in the module?<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">If you want to get help with the syntax and examples from the AzLogDcrLogIngestPS module, you can write get-module<\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code>PS  get-command -module AzLogDcrIngestPS\n\nCommandType     Name                                               Version    Source                                                                         \n-----------     ----                                               -------    ------                                                                         \nFunction        Add-CollectionTimeToAllEntriesInArray              1.1.17     AzLogDcrIngestPS                                                               \nFunction        Add-ColumnDataToAllEntriesInArray                  1.1.17     AzLogDcrIngestPS                                                               \nFunction        Build-DataArrayToAlignWithSchema                   1.1.17     AzLogDcrIngestPS                                                               \nFunction        CheckCreateUpdate-TableDcr-Structure               1.1.17     AzLogDcrIngestPS                                                               \nFunction        Convert-CimArrayToObjectFixStructure               1.1.17     AzLogDcrIngestPS                                                               \nFunction        Convert-PSArrayToObjectFixStructure                1.1.17     AzLogDcrIngestPS                                                               \nFunction        CreateUpdate-AzDataCollectionRuleLogIngestCusto... 1.1.17     AzLogDcrIngestPS                                                               \nFunction        CreateUpdate-AzLogAnalyticsCustomLogTableDcr       1.1.17     AzLogDcrIngestPS                                                               \nFunction        Delete-AzDataCollectionRules                       1.1.17     AzLogDcrIngestPS                                                               \nFunction        Delete-AzLogAnalyticsCustomLogTables               1.1.17     AzLogDcrIngestPS                                                               \nFunction        Filter-ObjectExcludeProperty                       1.1.17     AzLogDcrIngestPS                                                               \nFunction        Get-AzAccessTokenManagement                        1.1.17     AzLogDcrIngestPS                                                               \nFunction        Get-AzDceListAll                                   1.1.17     AzLogDcrIngestPS                                                               \nFunction        Get-AzDcrDceDetails                                1.1.17     AzLogDcrIngestPS                                                               \nFunction        Get-AzDataCollectionRuleTransformKql               1.1.17     AzLogDcrIngestPS                                                               \nFunction        Get-AzDcrListAll                                   1.1.17     AzLogDcrIngestPS                                                               \nFunction        Get-AzLogAnalyticsTableAzDataCollectionRuleStatus  1.1.17     AzLogDcrIngestPS                                                               \nFunction        Get-ObjectSchemaAsArray                            1.1.17     AzLogDcrIngestPS                                                               \nFunction        Get-ObjectSchemaAsHash                             1.1.17     AzLogDcrIngestPS                                                               \nFunction        Post-AzLogAnalyticsLogIngestCustomLogDcrDce        1.1.17     AzLogDcrIngestPS                                                               \nFunction        Post-AzLogAnalyticsLogIngestCustomLogDcrDce-Output 1.1.17     AzLogDcrIngestPS                                                               \nFunction        Update-AzDataCollectionRuleDceEndpoint             1.1.17     AzLogDcrIngestPS                                                               \nFunction        Update-AzDataCollectionRuleResetTransformKqlDef... 1.1.17     AzLogDcrIngestPS                                                               \nFunction        Update-AzDataCollectionRuleTransformKql            1.1.17     AzLogDcrIngestPS                                                               \nFunction        ValidateFix-AzLogAnalyticsTableSchemaColumnNames   1.1.17     AzLogDcrIngestPS                                                               \n\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h1 class=\"wp-block-heading\">How can I get access to the help, parameters, syntax, examples &#8211; using get-help ?<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Get help with a specific cmdlet &#8211; get-help Add-CollectionTimeToAllEntriesInArray -full<\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code>get-help Add-CollectionTimeToAllEntriesInArray -full\n\nNAME\n    Add-CollectionTimeToAllEntriesInArray\n    \nSYNOPSIS\n    Add property CollectionTime (based on current time) to all entries on the object\n    \n    \nSYNTAX\n    Add-CollectionTimeToAllEntriesInArray &#91;-Data] &lt;Array&gt; &#91;&lt;CommonParameters&gt;]\n    \n    \nDESCRIPTION\n    Gives capability to do proper searching in queries to find latest set of records with same collection time\n    Time Generated cannot be used when you are sending data in batches, as TimeGenerated will change\n    An example where this is important is a complete list of applications for a computer. We want all applications to\n    show up when queriying for the latest data\n    \n\nPARAMETERS\n    -Data &lt;Array&gt;\n        Object to modify\n        \n        Required?                    true\n        Position?                    1\n        Default value                \n        Accept pipeline input?       false\n        Accept wildcard characters?  false\n        \n    &lt;CommonParameters&gt;\n        This cmdlet supports the common parameters: Verbose, Debug,\n        ErrorAction, ErrorVariable, WarningAction, WarningVariable,\n        OutBuffer, PipelineVariable, and OutVariable. For more information, see \n        about_CommonParameters (https:\/go.microsoft.com\/fwlink\/?LinkID=113216). \n    \nINPUTS\n    None. You cannot pipe objects\n    \n    \nOUTPUTS\n    Updated object with CollectionTime\n    \n    \n    -------------------------- EXAMPLE 1 --------------------------\n    \n    PS C:\\&gt;#-------------------------------------------------------------------------------------------\n    \n    # Variables\n    #-------------------------------------------------------------------------------------------\n    $Verbose                   = $true  # $true or $false\n    \n    #-------------------------------------------------------------------------------------------\n    # Collecting data (in)\n    #-------------------------------------------------------------------------------------------\n    $DNSName                   = (Get-CimInstance win32_computersystem).DNSHostName +\".\" + (Get-CimInstance win32_computersystem).Domain\n    $ComputerName              = (Get-CimInstance win32_computersystem).DNSHostName\n    &#91;datetime]$CollectionTime  = ( Get-date (&#91;datetime]::Now.ToUniversalTime()) -format \"yyyy-MM-ddTHH:mm:ssK\" )\n    \n    $UserLoggedOnRaw           = Get-Process -IncludeUserName -Name explorer | Select-Object UserName -Unique\n    $UserLoggedOn              = $UserLoggedOnRaw.UserName\n    \n    $DataVariable = Get-CimInstance -ClassName Win32_Processor | Select-Object -ExcludeProperty \"CIM*\"\n    \n    #-------------------------------------------------------------------------------------------\n    # Preparing data structure\n    #-------------------------------------------------------------------------------------------\n    $DataVariable = Convert-CimArrayToObjectFixStructure -data $DataVariable -Verbose:$Verbose\n    $DataVariable\n    \n    # add CollectionTime to existing array\n    $DataVariable = Add-CollectionTimeToAllEntriesInArray -Data $DataVariable -Verbose:$Verbose\n    $DataVariable\n    \n    #-------------------------------------------------------------------------------------------\n    # Output\n    #-------------------------------------------------------------------------------------------\n    \n    VERBOSE:   Adding CollectionTime to all entries in array .... please wait !\n    Caption                                 : Intel64 Family 6 Model 165 Stepping 5\n    Description                             : Intel64 Family 6 Model 165 Stepping 5\n    InstallDate                             : \n    Name                                    : Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz\n    Status                                  : OK\n    Availability                            : 3\n    ConfigManagerErrorCode                  : \n    ConfigManagerUserConfig                 : \n    CreationClassName                       : Win32_Processor\n    DeviceID                                : CPU0\n    ErrorCleared                            : \n    ErrorDescription                        : \n    LastErrorCode                           : \n    PNPDeviceID                             : \n    PowerManagementCapabilities             : \n    PowerManagementSupported                : False\n    StatusInfo                              : 3\n    SystemCreationClassName                 : Win32_ComputerSystem\n    SystemName                              : STRV-MOK-DT-02\n    AddressWidth                            : 64\n    CurrentClockSpeed                       : 2904\n    DataWidth                               : 64\n    Family                                  : 198\n    LoadPercentage                          : 1\n    MaxClockSpeed                           : 2904\n    OtherFamilyDescription                  : \n    Role                                    : CPU\n    Stepping                                : \n    UniqueId                                : \n    UpgradeMethod                           : 1\n    Architecture                            : 9\n    AssetTag                                : To Be Filled By O.E.M.\n    Characteristics                         : 252\n    CpuStatus                               : 1\n    CurrentVoltage                          : 8\n    ExtClock                                : 100\n    L2CacheSize                             : 2048\n    L2CacheSpeed                            : \n    L3CacheSize                             : 16384\n    L3CacheSpeed                            : 0\n    Level                                   : 6\n    Manufacturer                            : GenuineIntel\n    NumberOfCores                           : 8\n    NumberOfEnabledCore                     : 8\n    NumberOfLogicalProcessors               : 16\n    PartNumber                              : To Be Filled By O.E.M.\n    ProcessorId                             : BFEBFBFF000A0655\n    ProcessorType                           : 3\n    Revision                                : \n    SecondLevelAddressTranslationExtensions : False\n    SerialNumber                            : To Be Filled By O.E.M.\n    SocketDesignation                       : U3E1\n    ThreadCount                             : 16\n    Version                                 : \n    VirtualizationFirmwareEnabled           : False\n    VMMonitorModeExtensions                 : False\n    VoltageCaps                             : \n    PSComputerName                          : \n    CollectionTime                          : 12-03-2023 16:08:33\n    \n    \n    \n    \n    \nRELATED LINKS\n    https:&#47;&#47;github.com\/KnudsenMorten\/AzLogDcrIngestPS\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>If you are sending data using HTTP Data Collector API (REST) today, you should continue reading, as this API will &#8230; <\/p>\n<p class=\"read-more-container\"><a title=\"AzLogDcrIngestPS &#8211; tips &#038; tricks sending data via Azure Pipeline, Azure Log Ingestion API, Azure Data Collections into Azure LogAnalytics\" class=\"read-more button\" href=\"https:\/\/mortenknudsen.net\/?p=1499#more-1499\" aria-label=\"Read more about AzLogDcrIngestPS &#8211; tips &#038; tricks sending data via Azure Pipeline, Azure Log Ingestion API, Azure Data Collections into Azure LogAnalytics\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":2192,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"ngg_post_thumbnail":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[147,55,129,131,128,54,143,57,28,136,137,142,133,134,60,132,58],"tags":[],"class_list":["post-1499","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azlogdcringestps","category-azure","category-azure-data-collection-rules","category-azure-data-ingestion-pipeline","category-azure-log-ingestion-api","category-azure-loganalytics","category-azure-logging","category-azure-security","category-community","category-data-transformation","category-dcr","category-logging","category-microsoft-security","category-mvpbuzz","category-scripting","category-security","category-sentinel","infinite-scroll-item","resize-featured-image"],"featured_image_src":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/datamanipulation-1.png","author_info":{"display_name":"Morten Knudsen","author_link":"https:\/\/mortenknudsen.net\/?author=1"},"jetpack_featured_media_url":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/datamanipulation-1.png","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1499","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1499"}],"version-history":[{"count":10,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1499\/revisions"}],"predecessor-version":[{"id":2275,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1499\/revisions\/2275"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/media\/2192"}],"wp:attachment":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1499"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1499"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1499"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}