AzLogDcrIngestPS – how to do data manipulation before sending data via Azure Pipeline, Log ingestion API & Azure Data Collection Rules into Azure LogAnalytics ?

If you are sending data using HTTP Data Collector API (REST) today, you should continue reading, as this API will be deprecated, as part of the transition to Log ingestion API using Azure Data Collection Rules, Azure Pipeline, Azure LogAnalytics custom tables (v2).

As you can see from the illustration more components (DCR, DCR, Pipeline, Schema) are added, which also increases the complexity.

I have built a Powershell module, AzLogDcrIngestPS which will ease the steps, if you want to send any data to Azure LogAnalytics custom logs (v2) – using the new features of Azure Log Ingestion PipelineAzure Data Colection Rules & Log Ingestion API.

AzLogDcrIngestPS includes 25 functions dealing with:

  • data manipulation before sending data in (7 functions)
  • table / dcr / schema / transformation management (13 functions)
  • data upload using Azure Log Ingestion Pipeline / Log Ingestion API (4 functions)
  • support/security (1 function)

This blog post will go in more details about how you can manipulate your data using functions in AzLogDcrIngestPS so data can be uploaded into Azure LogAnalytics using Log Ingestion API.

If you want to get started or want more information, check out this blog-post

Understanding your data

Before sending data into Azure LogAnalytics, it is obvious that you want to ensure your data is containing valuable data and structured so you can access them afterwards.

A good naming convention is also important, as data must be uploaded based upon a schema, both for DCR and LogAnalytics. LogAnalytics and DCR have prohibited properties (column-names in table).

In case data is coming from sources like CIM, WMI, etc. they will contain information, where I typically strip the extra data as it is just “noice”.


If you want to see the functions in action, check out the video below

Video 2m 19s – Data manipulation

See AzLogDcrIngestPS functions in action 🙂

I have provided 4 demos for you to try. I have also provided videos for you to check out.

Alternatively, I have built a cool showcase – ClientInspector (v2)free for the community, where you can see how you can use the funtions from the AzLogDcrIngestPS module.


ClientInspector can bring back data from your clients using Azure Log Ingestion PipelineAzure Data Collection RulesAzure LogAnalytics; view them with Azure Monitor & Azure Dashboards â€“ and get “drift-alerts” using Microsoft Sentinel. it includes tons of great information and dashboards to see if you are in control with your clients – or something is drifting from desired state.

Data manipulation functions

AzLogDcrIngestPS module provides 7 functions shown below.

Function nameSynopsis
Add-CollectionTimeToAllEntriesInArrayAdd property CollectionTime (based on current time) to all entries on the object
Add-ColumnDataToAllEntriesInArrayAdds up to 3 extra columns and data to the object
Build-DataArrayToAlignWithSchemaRebuilds the source object to match modified schema structure – used after usage of ValidateFix-AzLogAnalyticsTableSchemaColumnNames
Convert-CimArrayToObjectFixStructureConverts CIM array and remove CIM class information
Convert-PSArrayToObjectFixStructureConverts PS array and remove PS class information
Filter-ObjectExcludePropertyRemoves columns from the object which is considered “noice” and shouldn’t be send to logs
ValidateFix-AzLogAnalyticsTableSchemaColumnNamesValidates the column names in the schema are valid according the requirement for LogAnalytics tables.
Fixes any issues by rebuild the source object

How can I get access to the help, parameters, syntax, examples – using get-help ?

Get help with a specific cmdlet with the command

get-help Add-CollectionTimeToAllEntriesInArray -full

PS> get-help Add-CollectionTimeToAllEntriesInArray -full

    Add property CollectionTime (based on current time) to all entries on the object
    Add-CollectionTimeToAllEntriesInArray [-Data] <Array> [<CommonParameters>]
    Gives capability to do proper searching in queries to find latest set of records with same collection time
    Time Generated cannot be used when you are sending data in batches, as TimeGenerated will change
    An example where this is important is a complete list of applications for a computer. We want all applications to
    show up when queriying for the latest data

    -Data <Array>
        Object to modify
        Required?                    true
        Position?                    1
        Default value                
        Accept pipeline input?       false
        Accept wildcard characters?  false
        This cmdlet supports the common parameters: Verbose, Debug,
        ErrorAction, ErrorVariable, WarningAction, WarningVariable,
        OutBuffer, PipelineVariable, and OutVariable. For more information, see 
        about_CommonParameters (https:/ 
    None. You cannot pipe objects
    Updated object with CollectionTime
    -------------------------- EXAMPLE 1 --------------------------
    PS C:\>#-------------------------------------------------------------------------------------------
    # Variables
    $Verbose                   = $true  # $true or $false
    # Collecting data (in)
    $DNSName                   = (Get-CimInstance win32_computersystem).DNSHostName +"." + (Get-CimInstance win32_computersystem).Domain
    $ComputerName              = (Get-CimInstance win32_computersystem).DNSHostName
    [datetime]$CollectionTime  = ( Get-date ([datetime]::Now.ToUniversalTime()) -format "yyyy-MM-ddTHH:mm:ssK" )
    $UserLoggedOnRaw           = Get-Process -IncludeUserName -Name explorer | Select-Object UserName -Unique
    $UserLoggedOn              = $UserLoggedOnRaw.UserName
    $DataVariable = Get-CimInstance -ClassName Win32_Processor | Select-Object -ExcludeProperty "CIM*"
    # Preparing data structure
    $DataVariable = Convert-CimArrayToObjectFixStructure -data $DataVariable -Verbose:$Verbose
    # add CollectionTime to existing array
    $DataVariable = Add-CollectionTimeToAllEntriesInArray -Data $DataVariable -Verbose:$Verbose
    # Output
    VERBOSE:   Adding CollectionTime to all entries in array .... please wait !
    Caption                                 : Intel64 Family 6 Model 165 Stepping 5
    Description                             : Intel64 Family 6 Model 165 Stepping 5
    InstallDate                             : 
    Name                                    : Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz
    Status                                  : OK
    Availability                            : 3
    ConfigManagerErrorCode                  : 
    ConfigManagerUserConfig                 : 
    CreationClassName                       : Win32_Processor
    DeviceID                                : CPU0
    ErrorCleared                            : 
    ErrorDescription                        : 
    LastErrorCode                           : 
    PNPDeviceID                             : 
    PowerManagementCapabilities             : 
    PowerManagementSupported                : False
    StatusInfo                              : 3
    SystemCreationClassName                 : Win32_ComputerSystem
    SystemName                              : STRV-MOK-DT-02
    AddressWidth                            : 64
    CurrentClockSpeed                       : 2904
    DataWidth                               : 64
    Family                                  : 198
    LoadPercentage                          : 1
    MaxClockSpeed                           : 2904
    OtherFamilyDescription                  : 
    Role                                    : CPU
    Stepping                                : 
    UniqueId                                : 
    UpgradeMethod                           : 1
    Architecture                            : 9
    AssetTag                                : To Be Filled By O.E.M.
    Characteristics                         : 252
    CpuStatus                               : 1
    CurrentVoltage                          : 8
    ExtClock                                : 100
    L2CacheSize                             : 2048
    L2CacheSpeed                            : 
    L3CacheSize                             : 16384
    L3CacheSpeed                            : 0
    Level                                   : 6
    Manufacturer                            : GenuineIntel
    NumberOfCores                           : 8
    NumberOfEnabledCore                     : 8
    NumberOfLogicalProcessors               : 16
    PartNumber                              : To Be Filled By O.E.M.
    ProcessorId                             : BFEBFBFF000A0655
    ProcessorType                           : 3
    Revision                                : 
    SecondLevelAddressTranslationExtensions : False
    SerialNumber                            : To Be Filled By O.E.M.
    SocketDesignation                       : U3E1
    ThreadCount                             : 16
    Version                                 : 
    VirtualizationFirmwareEnabled           : False
    VMMonitorModeExtensions                 : False
    VoltageCaps                             : 
    PSComputerName                          : 
    CollectionTime                          : 12-03-2023 16:08:33

Download latest version

You can download latest version of AzLogDcrIngestPS here – or install from Powershell Gallery:

Install AzLogDcringestPS from Powershell Gallery

install-module AzLogDcrIngestPS

Download AzLogDcringestPS module from this Github repositry

1 thought on “AzLogDcrIngestPS – how to do data manipulation before sending data via Azure Pipeline, Log ingestion API & Azure Data Collection Rules into Azure LogAnalytics ?”

Leave a Reply