If you are sending data using HTTP Data Collector API (REST) today, you should continue reading, as this API will be deprecated, as part of the transition to Log ingestion API using Azure Data Collection Rules, Azure Pipeline, Azure LogAnalytics custom tables (v2).
As you can see from the illustration more components (DCR, DCR, Pipeline, Schema) are added, which also increases the complexity.
I have built a Powershell module, AzLogDcrIngestPS which will ease the steps, if you want to send any data to Azure LogAnalytics custom logs (v2) – using the new features of Azure Log Ingestion Pipeline, Azure Data Colection Rules & Log Ingestion API.
AzLogDcrIngestPS includes 25 functions dealing with:
- data manipulation before sending data in (7 functions)
- table / dcr / schema / transformation management (13 functions)
- data upload using Azure Log Ingestion Pipeline / Log Ingestion API (4 functions)
- support/security (1 function)
This blog post will go in more details about how you can manipulate your data using functions in AzLogDcrIngestPS so data can be uploaded into Azure LogAnalytics using Log Ingestion API.
If you want to get started or want more information, check out this blog-post
Understanding your data
Before sending data into Azure LogAnalytics, it is obvious that you want to ensure your data is containing valuable data and structured so you can access them afterwards.
A good naming convention is also important, as data must be uploaded based upon a schema, both for DCR and LogAnalytics. LogAnalytics and DCR have prohibited properties (column-names in table).
In case data is coming from sources like CIM, WMI, etc. they will contain information, where I typically strip the extra data as it is just “noice”.
If you want to see the functions in action, check out the video below
Video 2m 19s – Data manipulation
See AzLogDcrIngestPS functions in action 🙂
I have provided 4 demos for you to try. I have also provided videos for you to check out.
Alternatively, I have built a cool showcase – ClientInspector (v2), free for the community, where you can see how you can use the funtions from the AzLogDcrIngestPS module.
ClientInspector can bring back data from your clients using Azure Log Ingestion Pipeline, Azure Data Collection Rules, Azure LogAnalytics; view them with Azure Monitor & Azure Dashboards – and get “drift-alerts” using Microsoft Sentinel. it includes tons of great information and dashboards to see if you are in control with your clients – or something is drifting from desired state.
Data manipulation functions
AzLogDcrIngestPS module provides 7 functions shown below.
| Function name | Synopsis |
|---|---|
| Add-CollectionTimeToAllEntriesInArray | Add property CollectionTime (based on current time) to all entries on the object |
| Add-ColumnDataToAllEntriesInArray | Adds up to 3 extra columns and data to the object |
| Build-DataArrayToAlignWithSchema | Rebuilds the source object to match modified schema structure – used after usage of ValidateFix-AzLogAnalyticsTableSchemaColumnNames |
| Convert-CimArrayToObjectFixStructure | Converts CIM array and remove CIM class information |
| Convert-PSArrayToObjectFixStructure | Converts PS array and remove PS class information |
| Filter-ObjectExcludeProperty | Removes columns from the object which is considered “noice” and shouldn’t be send to logs |
| ValidateFix-AzLogAnalyticsTableSchemaColumnNames | Validates the column names in the schema are valid according the requirement for LogAnalytics tables. Fixes any issues by rebuild the source object |
How can I get access to the help, parameters, syntax, examples – using get-help ?
Get help with a specific cmdlet with the command
get-help Add-CollectionTimeToAllEntriesInArray -full
PS> get-help Add-CollectionTimeToAllEntriesInArray -full
NAME
Add-CollectionTimeToAllEntriesInArray
SYNOPSIS
Add property CollectionTime (based on current time) to all entries on the object
SYNTAX
Add-CollectionTimeToAllEntriesInArray [-Data] <Array> [<CommonParameters>]
DESCRIPTION
Gives capability to do proper searching in queries to find latest set of records with same collection time
Time Generated cannot be used when you are sending data in batches, as TimeGenerated will change
An example where this is important is a complete list of applications for a computer. We want all applications to
show up when queriying for the latest data
PARAMETERS
-Data <Array>
Object to modify
Required? true
Position? 1
Default value
Accept pipeline input? false
Accept wildcard characters? false
<CommonParameters>
This cmdlet supports the common parameters: Verbose, Debug,
ErrorAction, ErrorVariable, WarningAction, WarningVariable,
OutBuffer, PipelineVariable, and OutVariable. For more information, see
about_CommonParameters (https:/go.microsoft.com/fwlink/?LinkID=113216).
INPUTS
None. You cannot pipe objects
OUTPUTS
Updated object with CollectionTime
-------------------------- EXAMPLE 1 --------------------------
PS C:\>#-------------------------------------------------------------------------------------------
# Variables
#-------------------------------------------------------------------------------------------
$Verbose = $true # $true or $false
#-------------------------------------------------------------------------------------------
# Collecting data (in)
#-------------------------------------------------------------------------------------------
$DNSName = (Get-CimInstance win32_computersystem).DNSHostName +"." + (Get-CimInstance win32_computersystem).Domain
$ComputerName = (Get-CimInstance win32_computersystem).DNSHostName
[datetime]$CollectionTime = ( Get-date ([datetime]::Now.ToUniversalTime()) -format "yyyy-MM-ddTHH:mm:ssK" )
$UserLoggedOnRaw = Get-Process -IncludeUserName -Name explorer | Select-Object UserName -Unique
$UserLoggedOn = $UserLoggedOnRaw.UserName
$DataVariable = Get-CimInstance -ClassName Win32_Processor | Select-Object -ExcludeProperty "CIM*"
#-------------------------------------------------------------------------------------------
# Preparing data structure
#-------------------------------------------------------------------------------------------
$DataVariable = Convert-CimArrayToObjectFixStructure -data $DataVariable -Verbose:$Verbose
$DataVariable
# add CollectionTime to existing array
$DataVariable = Add-CollectionTimeToAllEntriesInArray -Data $DataVariable -Verbose:$Verbose
$DataVariable
#-------------------------------------------------------------------------------------------
# Output
#-------------------------------------------------------------------------------------------
VERBOSE: Adding CollectionTime to all entries in array .... please wait !
Caption : Intel64 Family 6 Model 165 Stepping 5
Description : Intel64 Family 6 Model 165 Stepping 5
InstallDate :
Name : Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz
Status : OK
Availability : 3
ConfigManagerErrorCode :
ConfigManagerUserConfig :
CreationClassName : Win32_Processor
DeviceID : CPU0
ErrorCleared :
ErrorDescription :
LastErrorCode :
PNPDeviceID :
PowerManagementCapabilities :
PowerManagementSupported : False
StatusInfo : 3
SystemCreationClassName : Win32_ComputerSystem
SystemName : STRV-MOK-DT-02
AddressWidth : 64
CurrentClockSpeed : 2904
DataWidth : 64
Family : 198
LoadPercentage : 1
MaxClockSpeed : 2904
OtherFamilyDescription :
Role : CPU
Stepping :
UniqueId :
UpgradeMethod : 1
Architecture : 9
AssetTag : To Be Filled By O.E.M.
Characteristics : 252
CpuStatus : 1
CurrentVoltage : 8
ExtClock : 100
L2CacheSize : 2048
L2CacheSpeed :
L3CacheSize : 16384
L3CacheSpeed : 0
Level : 6
Manufacturer : GenuineIntel
NumberOfCores : 8
NumberOfEnabledCore : 8
NumberOfLogicalProcessors : 16
PartNumber : To Be Filled By O.E.M.
ProcessorId : BFEBFBFF000A0655
ProcessorType : 3
Revision :
SecondLevelAddressTranslationExtensions : False
SerialNumber : To Be Filled By O.E.M.
SocketDesignation : U3E1
ThreadCount : 16
Version :
VirtualizationFirmwareEnabled : False
VMMonitorModeExtensions : False
VoltageCaps :
PSComputerName :
CollectionTime : 12-03-2023 16:08:33
RELATED LINKS
https://github.com/KnudsenMorten/AzLogDcrIngestPSDownload latest version
You can download latest version of AzLogDcrIngestPS here – or install from Powershell Gallery:
Install AzLogDcringestPS from Powershell Gallery
install-module AzLogDcrIngestPS
1 thought on “AzLogDcrIngestPS – how to do data manipulation before sending data via Azure Pipeline, Log ingestion API & Azure Data Collection Rules into Azure LogAnalytics ?”