{"id":1449,"date":"2023-04-02T18:36:02","date_gmt":"2023-04-02T17:36:02","guid":{"rendered":"https:\/\/mortenknudsen.net\/?p=1449"},"modified":"2023-04-13T21:00:33","modified_gmt":"2023-04-13T20:00:33","slug":"collecting-performance-data-using-azure-monitor-agent-vminsights-and-servicemap","status":"publish","type":"post","link":"https:\/\/mortenknudsen.net\/?p=1449","title":{"rendered":"Collecting Performance data using Azure Monitor Agent, VMInsights and ServiceMap"},"content":{"rendered":"\n

This blog will give you insight on how to setup collection of Performance and ServiceMap information<\/strong> from Linux<\/strong> and Windows devices<\/strong> using Azure Monitor Agent (AMA).<\/strong> This is also referred to as VMInsight<\/strong>.<\/p>\n\n\n\n

VM insights collects performance and connection metrics, computer and process inventory data, and health state information and forwards it to the Log Analytics workspace in Azure Monitor. <\/p>\n\n\n\n

You can use the data for example when you do migration planning, capacity analysis, discovery, and on-demand performance troubleshooting.<\/p>\n\n\n\n

Performance data<\/strong><\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

ServiceMap<\/strong><\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

This blog-post is part of a series of blog posts to master Azure logging in depth (overview)<\/a>. <\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

<\/p>\n\n\n\n

To get you started, you can find ARM-templates & scripts in my AzureLogLibrary (github)<\/a>. Details will be covered in the articles.<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n


<\/p>\n\n\n\n

<\/p>\n\n\n\n

Quick Links<\/h1>\n\n\n\n

How to start collecting performance & service map events using ARM-template ?<\/a>
How to setup in GUI<\/a>
High-level architecture \/ flow<\/a>
Layout of VMInsight DCR (JSON)<\/a>
Data transformation<\/a><\/p>\n\n\n\n

<\/p>\n\n\n\n

How to start collecting performance & service map events using ARM-template ?<\/h1>\n\n\n\n

To get you started, I have prepared ARM-templates on my Github<\/a><\/p>\n\n\n\n

VMInsight – Windows – Performance_ServiceMap<\/a><\/td><\/tr>
VMInsight – Linux – Performance_ServiceMap<\/a><\/td><\/tr>
VMInsight – Linux – Performance_Basic<\/a><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

I have also provided more documentation and powershell script to deploy more DCRs<\/a><\/p>\n\n\n\n

Lastly, I am also providing \u2018Deploy to Azure\u2019 shortcuts, based on the mentioned ARM-templates so you can deploy samples into your environment.<\/p>\n\n\n\n

<\/p>\n\n\n\n

VMInsight – Windows – Performance_ServiceMap<\/h2>\n\n\n\n\n \n<\/a>\n\n\n\n

<\/p>\n\n\n\n

VMInsight – Linux – Performance_ServiceMap<\/h2>\n\n\n\n\n \n<\/a>\n\n\n\n

<\/p>\n\n\n\n

VMInsight – Linux – Performance_Basic<\/h2>\n\n\n\n\n \n<\/a>\n\n\n\n


How to setup in GUI ?<\/h1>\n\n\n\n

If you prefer to setup using the GUI, you will only be able to collect basic performance counters – and not servicemap information. Please use the provided ARM-templates to setup a combi-collection – see the previous section.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

You don’t need to associate the DCR rule with any resources when you are just setting up the DCR rule. You can do this later.<\/p>\n\n\n\n

Some data sources require a data collection endpoint (txt-logs, iis-logs, log ingestion api). This is covered in another blog-post<\/a><\/p>\n\n\n\n

You can just click Next<\/p>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

ServiceMap information provides cool features, where you will be able to see which devices are talking with the server – and to which processes. You will see both inbound and outbound traffic.<\/p>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n


<\/p>\n\n\n\n

High-level architecture \/ flow<\/h1>\n\n\n\n

Data will be sent into the following Azure LogAnalytics standard tables:<\/p>\n\n\n\n

InsightsMetrics<\/td>Performance counters & health state (heartbeat) counters – separated into the following Namespaces:

Computer
LogicalDisk
Network
memory
Processer<\/td><\/tr>
VMBound
VMConnection<\/td>		These tables provide information about the connections for a machine (inbound and outbound), as well as the server ports that are open\/active on them.

ConnectionMetrics are also exposed via APIs that provide the means to obtain a specific metric during a time window.

TCP connections resulting from accepting<\/em> on a listening socket are inbound, while those created by connecting<\/em> to a given IP and port are outbound. The direction of a connection is represented by the Direction property, which can be set to either inbound<\/strong> or outbound<\/strong>.

Records in these tables are generated from data reported by the Dependency Agent.

Every record represents an observation over a 1-minute time interval.

The TimeGenerated property indicates the start of the time interval. Each record contains information to identify the respective entity, that is, connection or port, as well as metrics associated with that entity.

Currently, only network activity that occurs using TCP over IPv4 is reported.<\/td><\/tr>
VMProcess<\/td>Contains all processes and PIDs running on the device<\/td><\/tr>
VMComputer<\/td>Contains detailed information about the device<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

Layout of VMInsight DCR (JSON)<\/h1>\n\n\n\n

DCR contains the following dataSources<\/strong> section, if you want to collect vminsight data.<\/p>\n\n\n\n

 \"dataSources\": {\n            \"performanceCounters\": [\n                {\n                    \"streams\": [\n                        \"Microsoft-InsightsMetrics\"\n                    ],\n                    \"samplingFrequencyInSeconds\": 60,\n                    \"counterSpecifiers\": [\n                        \"\\\\VmInsights\\\\DetailedMetrics\"\n                    ],\n                    \"name\": \"VMInsightsPerfCounters\"\n                }\n        <\/code><\/pre>\n\n\n\n
{\n    \"properties\": {\n        \"description\": \"Data collection rule for VM Insights.\",\n        \"immutableId\": \"dcr-f25135f34aba4b3ca04ce623aeace6a1\",\n        \"dataSources\": {\n            \"performanceCounters\": [\n                {\n                    \"streams\": [\n                        \"Microsoft-InsightsMetrics\"\n                    ],\n                    \"samplingFrequencyInSeconds\": 60,\n                    \"counterSpecifiers\": [\n                        \"\\\\VmInsights\\\\DetailedMetrics\"\n                    ],\n                    \"name\": \"VMInsightsPerfCounters\"\n                }\n            ],\n            \"extensions\": [\n                {\n                    \"streams\": [\n                        \"Microsoft-ServiceMap\"\n                    ],\n                    \"extensionName\": \"DependencyAgent\",\n                    \"extensionSettings\": {},\n                    \"name\": \"DependencyAgentDataSource\"\n                }\n            ]\n        },\n        \"destinations\": {\n            \"logAnalytics\": [\n                {\n                    \"workspaceResourceId\": \"\/subscriptions\/fce4f282-fcc6-43fb-94d8-bf1701b862c3\/resourcegroups\/rg-logworkspaces\/providers\/microsoft.operationalinsights\/workspaces\/log-platform-management-srvnetworkcloud-p\",\n                    \"workspaceId\": \"b7d80924-d55d-4bf6-b2b3-9889301c7114\",\n                    \"name\": \"VMInsightsPerf-Logs-Dest\"\n                }\n            ]\n        },\n        \"dataFlows\": [\n            {\n                \"streams\": [\n                    \"Microsoft-InsightsMetrics\"\n                ],\n                \"destinations\": [\n                    \"VMInsightsPerf-Logs-Dest\"\n                ]\n            },\n            {\n                \"streams\": [\n                    \"Microsoft-ServiceMap\"\n                ],\n                \"destinations\": [\n                    \"VMInsightsPerf-Logs-Dest\"\n                ]\n            }\n        ],\n        \"provisioningState\": \"Succeeded\"\n    }<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
Data transformation<\/h1>\n\n\n\n
It doesn’t make much sense to do transformation of data related to servicemap or performance data. Instead I would recommend changing the frequency of how often you retrieve the data – by default every 60 sec.<\/p>\n\n\n\n
You will manage the frequency using the samplerFrequencyInSeconds<\/strong><\/p>\n\n\n\n
 \"samplingFrequencyInSeconds\": 60,\n                    \"counterSpecifiers\": [\n                        \"\\\\VmInsights\\\\DetailedMetrics\"<\/code><\/pre>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
This blog will give you insight on how to setup collection of Performance and ServiceMap information from Linux and Windows … <\/p>\n
Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":1897,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"ngg_post_thumbnail":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[140,55,59,129,131,61,128,54,143,141,57,139,136,138,137,142,134,146,60,132,58,145,135,144],"tags":[],"class_list":["post-1449","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ama","category-azure","category-azure-arc","category-azure-data-collection-rules","category-azure-data-ingestion-pipeline","category-azure-extensions","category-azure-log-ingestion-api","category-azure-loganalytics","category-azure-logging","category-azure-monitor-agent","category-azure-security","category-data-collection-endpoint","category-data-transformation","category-dce","category-dcr","category-logging","category-mvpbuzz","category-performance-collection","category-scripting","category-security","category-sentinel","category-servicemap","category-transformation","category-vminsight","infinite-scroll-item","resize-featured-image"],"featured_image_src":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/insight.png","author_info":{"display_name":"Morten Knudsen","author_link":"https:\/\/mortenknudsen.net\/?author=1"},"jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"jetpack_featured_media_url":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/insight.png","_links":{"self":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1449","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1449"}],"version-history":[{"count":29,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1449\/revisions"}],"predecessor-version":[{"id":2290,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1449\/revisions\/2290"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/media\/1897"}],"wp:attachment":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1449"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1449"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1449"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}