Recently, I was challenged to build a scalable, cloud native solution that should be used for monitoring of critical Windows Services on both Azure Native and Azure Arc servers. Solution should scale to +1000 servers.
Requirements included the concept of server maintenance mode (true/false); send mail & SMS notification when stopped & running; run every 5 min; easy to manage using tags and integrate with Azure Update Manager maintenance config for Patching (pre/post).
This blog shows how I built it using Azure Tags, Kusto Query against Azure Resource Graph and Azure LogAnalytics Change Tracking data; collected with Azure Monitor Agent using Azure Data Collection Rule and everything wrapped into an Azure Logic App.
Content
- Output: Service is stopped
- Output: Service is running
- Overview: Azure Logic Apps Engine
- Tags used on Azure Native & Azure Arc Servers
- Azure Logic App configuration
- Azure Data Collection Rules for Change Tracking Collection
- Azure LogAnalytics Workspace for Change Tracking
Output: Service is stopped
Output: Service is running
Overview: Azure Logic Apps Engine
Tags used on Azure Native & Azure Arc Servers
TagName | TagValues | Purpose |
MonitorMaintenanceModeActive | FALSE TRUE | If a server should be put into maintenance mode, you can set the value to TRUE. Then no alerts will arrive until status is changed back t FALSE again. You can also control this tag-value using azure Update Manager pre/post-events |
MonitorWindowsServices | Windows Service name, separated by , (comma) sample IISADMIN,Dfs,DFSR,DNS,ADSync,AADConnectProvisioningAgent | Lists the services that should be monitored |
MonitorContactMail | Mail address like mok@mortenknudsen.net | Value of it alerts group that should receive notifications |
MonitorContactSMS | Mobile number like +45 12345678 | Mobile number of recipient |
Azure Logic App configuration
The Logic app is defined by the following phases:
Step # | Step | Action |
1 | Recurrence | Runs every 5 min |
2 | Query stopped/started (list output format) | Checks if any service in scope has stopped (or started) during last 5 min. |
3 | Condition (output) from step 2 | If output, then it continues to step 4 |
4 | Query stopped/started (HTML output format) | Checks if any service in scope has stopped (or started) during last 5 min. It is 100% the same output as step 2, but it builds an HTML output with tables |
5 | Send mail | Here mail it sent to the variable ContactMail |
6 | Send SMS (optional) | Here mail it sent to the variable ContactSMS |
Step 1 – Recurrence
Step 2 – Query stopped/started (list output format)
Step 2 – Query – Detect Windows Service Stopped
arg("").resources
| where type in ('microsoft.compute/virtualmachines','microsoft.hybridcompute/machines')
| extend ContactMail = tostring(tags["MonitorContactMail"])
| extend ContactSMS = tostring(tags["MonitorContactSMS"])
| extend MonitorMaintenanceModeActive = tostring(tags["MonitorMaintenanceModeActive"])
| extend PatchGroup = tostring(tags["PatchGroup"])
| extend PatchActive = tostring(tags["PatchActive"])
| mvexpand tags
| extend Name = toupper(name)
| extend tagKey = tostring(bag_keys(tags)[0])
| extend tagValue = split(tostring(tags[tagKey]), ",")
| extend tagValueStr = tostring(tags[tagKey])
| where tagKey in ("MonitorContactMail","MonitorContactSMS","MonitorMaintenanceModeActive","MonitorWindowsServices","PatchActive","PatchGroup","Environment")
| project Name, tagKey, tagValue, tagValueStr, type, location, resourceGroup, subscriptionId, id, ContactMail, ContactSMS, MonitorMaintenanceModeActive, PatchGroup, PatchActive
| join kind=inner (
ConfigurationChange
| extend Name = toupper(Computer)
) on $left.Name == $right.Name
| where TimeGenerated > ago(5m)
| where MonitorMaintenanceModeActive =~ "FALSE"
| where ConfigChangeType == "WindowsServices"
| where SvcChangeType == "State"
| where SvcState == "Stopped"
| where SvcPreviousState == "Running"
| where tagKey =~ "MonitorWindowsServices"
| where SvcStartupType == "Auto"
| sort by TimeGenerated desc
| mv-apply tagValue on (
where SvcName == (tagValue)
)
| project Name, TimeGenerated, SvcName, SvcDisplayName, SvcState, SvcPreviousState, SvcStartupType, tagKey, tagValue, tagValueStr, ContactMail, ContactSMS, MonitorMaintenanceModeActive, PatchGroup, PatchActive
Step 2 – Query – Detect Windows Service Running
arg("").resources
| where type in ('microsoft.compute/virtualmachines','microsoft.hybridcompute/machines')
| extend ContactMail = tostring(tags["MonitorContactMail"])
| extend ContactSMS = tostring(tags["MonitorContactSMS"])
| extend MonitorMaintenanceModeActive = tostring(tags["MonitorMaintenanceModeActive"])
| extend PatchGroup = tostring(tags["PatchGroup"])
| extend PatchActive = tostring(tags["PatchActive"])
| mvexpand tags
| extend Name = toupper(name)
| extend tagKey = tostring(bag_keys(tags)[0])
| extend tagValue = split(tostring(tags[tagKey]), ",")
| extend tagValueStr = tostring(tags[tagKey])
| where tagKey in ("MonitorContactMail","MonitorContactSMS","MonitorMaintenanceModeActive","MonitorWindowsServices","PatchActive","PatchGroup","Environment")
| project Name, tagKey, tagValue, tagValueStr, type, location, resourceGroup, subscriptionId, id, ContactMail, ContactSMS, MonitorMaintenanceModeActive, PatchGroup, PatchActive
| join kind=inner (
ConfigurationChange
| extend Name = toupper(Computer)
) on $left.Name == $right.Name
| where TimeGenerated > ago(5m)
| where MonitorMaintenanceModeActive =~ "FALSE"
| where ConfigChangeType == "WindowsServices"
| where SvcChangeType == "State"
| where SvcState == "Running"
| where SvcPreviousState == "Stopped"
| where tagKey =~ "MonitorWindowsServices"
| where SvcStartupType == "Auto"
| sort by TimeGenerated desc
| mv-apply tagValue on (
where SvcName == (tagValue)
)
| project Name, TimeGenerated, SvcName, SvcDisplayName, SvcState, SvcPreviousState, SvcStartupType, tagKey, tagValue, tagValueStr, ContactMail, ContactSMS, MonitorMaintenanceModeActive, PatchGroup, PatchActive
Step 3 – Conditions
Validate code
"type": "If",
"expression": {
"and": [
{
"equals": [
"@length(body('Detect_Windows_Service_Stopped_-_LIST')?['value'])",
0
]
}
]
Step 4 – Query stopped/started (HTML output format)
Use same queries as earlier (stopped/running)
Step 5 – Send Mail
Final layout
Validate that Azure Logic Apps runs every 5 min
Azure Logic Apps (Code View)
Azure Logic Apps (Code View)
Code View
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"contentVersion": "1.0.0.0",
"triggers": {
"Recurrence": {
"type": "Recurrence"
}
},
"actions": {
"Detect_Windows_Service_Stopped_-_LIST": {
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['azuremonitorlogs']['connectionId']"
}
},
"method": "post",
"path": "/queryDataV2"
},
"runAfter": {}
},
"Condition_(stopped)": {
"type": "If",
"actions": {},
"else": {
"actions": {
"Detect_Windows_Service_Stopped_-_HTML": {
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['azuremonitorlogs']['connectionId']"
}
},
"method": "post",
"path": "/visualizeQuery"
}
},
"For_each": {
"type": "Foreach",
"actions": {},
"runAfter": {
"Detect_Windows_Service_Stopped_-_HTML": [
"Succeeded"
]
}
}
}
},
"runAfter": {
"Detect_Windows_Service_Stopped_-_LIST": [
"Succeeded"
]
}
},
"Detect_Windows_Service_Running_-_LIST": {
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['azuremonitorlogs']['connectionId']"
}
},
"method": "post",
"path": "/queryDataV2"
},
"runAfter": {}
},
"Condition_(runnng)": {
"type": "If",
"actions": {},
"else": {
"actions": {
"Detect_Windows_Service_Started_-_HTML": {
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['azuremonitorlogs']['connectionId']"
}
},
"method": "post",
"path": "/visualizeQuery"
}
},
"For_each_1": {
"type": "Foreach",
"actions": {},
"runAfter": {
"Detect_Windows_Service_Started_-_HTML": [
"Succeeded"
]
}
}
}
},
"runAfter": {
"Detect_Windows_Service_Running_-_LIST": [
"Succeeded"
]
}
}
},
"outputs": {},
"parameters": {
"$connections": {
"type": "Object",
"defaultValue": {}
}
}
},
"parameters": {
"$connections": {
"value": {
"azuremonitorlogs": {
"id": "/subscriptions/xxxxxxxxxx3-12e2bf9fbec0/providers/Microsoft.Web/locations/westeurope/managedApis/azuremonitorlogs",
"connectionId": "/subscriptions/xxxxxxxxxxx-12e2bf9fbec0/resourceGroups/rg-demo/providers/Microsoft.Web/connections/azuremonitorlogs-3",
"connectionName": "azuremonitorlogs-3",
"connectionProperties": {
"authentication": {
"type": "ManagedServiceIdentity"
}
}
},
"office365-1": {
"id": "/subscriptions/1xxxxxxxxxx-12e2bf9fbec0/providers/Microsoft.Web/locations/westeurope/managedApis/office365",
"connectionId": "/subscriptions/xxxxxxxxx3-12e2bf9fbec0/resourceGroups/rg-demo/providers/Microsoft.Web/connections/office365-1",
"connectionName": "office365-1"
}
}
}
}
}
Azure Data Collection Rules for Change Tracking Collection
Windows service collection runs every 60 sec. You can control this frequency by adjusting this value.
“servicesSettings”: {
“serviceCollectionFrequency”: 60
Don’t forget to associate the servers, where you want to collect Windows Service Change Tracking information as shown below
Azure Data Collection Rules Template (JSON)
Azure Data Collection Rules Template (JSON). Used with Custom Deployment
dcr-operation-windows-changetracking.json
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"WorkspaceResourceId": {
"type": "String",
"metadata": {
"description": "LogAnalytics Workspace Resource ID"
}
},
"WorkspaceLocation": {
"type": "String",
"metadata": {
"description": "LogAnalytics Location (e.g. westeurope)"
}
},
"DcrName": {
"type": "String",
"metadata": {
"description": "Data Collection Rule name"
}
},
"DcrResourceGroup": {
"type": "String",
"metadata": {
"description": "Data Collection Rule resource group"
}
}
},
"variables": {},
"resources": [
{
"type": "microsoft.resources/deployments",
"name": "[parameters('DcrName')]",
"apiVersion": "2020-08-01",
"properties": {
"mode": "Incremental",
"template": {
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"resources": [
{
"type": "Microsoft.Insights/dataCollectionRules",
"apiVersion": "2021-04-01",
"name": "[parameters('DcrName')]",
"location": "[parameters('WorkspaceLocation')]",
"properties": {
"description": "Data collection rule for ChangeTracking",
"dataSources": {
"extensions": [
{
"streams": [
"Microsoft-ConfigurationChange",
"Microsoft-ConfigurationChangeV2",
"Microsoft-ConfigurationData"
],
"extensionName": "ChangeTracking-Windows",
"extensionSettings": {
"enableFiles": false,
"enableSoftware": false,
"enableRegistry": false,
"enableServices": true,
"enableInventory": false,
"registrySettings": {
"registryCollectionFrequency": 3000,
"registryInfo": [
{
"name": "Registry_1",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\Software\\Microsoft\\Windows\\CurrentVersion\\Group Policy\\Scripts\\Startup",
"valueName": ""
},
{
"name": "Registry_2",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\Software\\Microsoft\\Windows\\CurrentVersion\\Group Policy\\Scripts\\Shutdown",
"valueName": ""
},
{
"name": "Registry_3",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\SOFTWARE\\Wow6432Node\\Microsoft\\Windows\\CurrentVersion\\Run",
"valueName": ""
},
{
"name": "Registry_4",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Active Setup\\Installed Components",
"valueName": ""
},
{
"name": "Registry_5",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\Software\\Classes\\Directory\\ShellEx\\ContextMenuHandlers",
"valueName": ""
},
{
"name": "Registry_6",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\Software\\Classes\\Directory\\Background\\ShellEx\\ContextMenuHandlers",
"valueName": ""
},
{
"name": "Registry_7",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\Software\\Classes\\Directory\\Shellex\\CopyHookHandlers",
"valueName": ""
},
{
"name": "Registry_8",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\Software\\Microsoft\\Windows\\CurrentVersion\\Explorer\\ShellIconOverlayIdentifiers",
"valueName": ""
},
{
"name": "Registry_9",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\Software\\Wow6432Node\\Microsoft\\Windows\\CurrentVersion\\Explorer\\ShellIconOverlayIdentifiers",
"valueName": ""
},
{
"name": "Registry_10",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\Software\\Microsoft\\Windows\\CurrentVersion\\Explorer\\Browser Helper Objects",
"valueName": ""
},
{
"name": "Registry_11",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\Software\\Wow6432Node\\Microsoft\\Windows\\CurrentVersion\\Explorer\\Browser Helper Objects",
"valueName": ""
},
{
"name": "Registry_12",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\Software\\Microsoft\\Internet Explorer\\Extensions",
"valueName": ""
},
{
"name": "Registry_13",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\Software\\Wow6432Node\\Microsoft\\Internet Explorer\\Extensions",
"valueName": ""
},
{
"name": "Registry_14",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\Software\\Microsoft\\Windows NT\\CurrentVersion\\Drivers32",
"valueName": ""
},
{
"name": "Registry_15",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\Software\\Wow6432Node\\Microsoft\\Windows NT\\CurrentVersion\\Drivers32",
"valueName": ""
},
{
"name": "Registry_16",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Control\\Session Manager\\KnownDlls",
"valueName": ""
},
{
"name": "Registry_17",
"groupTag": "Recommended",
"enabled": false,
"recurse": true,
"description": "",
"keyName": "HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon\\Notify",
"valueName": ""
}
]
},
"fileSettings": {
"fileCollectionFrequency": 2700
},
"softwareSettings": {
"softwareCollectionFrequency": 1800
},
"inventorySettings": {
"inventoryCollectionFrequency": 36000
},
"servicesSettings": {
"serviceCollectionFrequency": 60
}
},
"name": "CTDataSource-Windows"
},
{
"streams": [
"Microsoft-ConfigurationChange",
"Microsoft-ConfigurationChangeV2",
"Microsoft-ConfigurationData"
],
"extensionName": "ChangeTracking-Linux",
"extensionSettings": {
"enableFiles": false,
"enableSoftware": false,
"enableRegistry": false,
"enableServices": true,
"enableInventory": false,
"fileSettings": {
"fileCollectionFrequency": 900,
"fileInfo": [
{
"name": "ChangeTrackingLinuxPath_default",
"enabled": true,
"destinationPath": "/etc/*.conf",
"useSudo": true,
"recurse": true,
"maxContentsReturnable": 5000000,
"pathType": "File",
"type": "File",
"links": "Follow",
"maxOutputSize": 500000,
"groupTag": "Recommended"
}
]
},
"softwareSettings": {
"softwareCollectionFrequency": 300
},
"inventorySettings": {
"inventoryCollectionFrequency": 36000
},
"servicesSettings": {
"serviceCollectionFrequency": 60
}
},
"name": "CTDataSource-Linux"
}
]
},
"destinations": {
"logAnalytics": [
{
"workspaceResourceId": "[parameters('workspaceResourceId')]",
"name": "Microsoft-CT-Dest"
}
]
},
"dataFlows": [
{
"streams": [
"Microsoft-ConfigurationChange",
"Microsoft-ConfigurationChangeV2",
"Microsoft-ConfigurationData"
],
"destinations": [
"Microsoft-CT-Dest"
]
}
]
}
}
]
}
},
"subscriptionId": "[split(parameters('WorkspaceResourceId'),'/')[2]]",
"resourceGroup": "[parameters('DcrResourceGroup')]"
}
]
}