{"id":1687,"date":"2023-04-02T08:12:51","date_gmt":"2023-04-02T07:12:51","guid":{"rendered":"https:\/\/mortenknudsen.net\/?p=1687"},"modified":"2024-09-14T14:25:08","modified_gmt":"2024-09-14T13:25:08","slug":"000-azure-logging-blog-overview","status":"publish","type":"post","link":"https:\/\/mortenknudsen.net\/?p=1687","title":{"rendered":"Master Azure Logging in depth"},"content":{"rendered":"\n

I am really passioned<\/strong> about the logging capabilities<\/strong> in M365 Defender<\/strong> and Azure<\/strong> with the power to bring data back from clients, servers, cloud and 3rd party systems<\/strong> – and getting cool valuable information out of the data<\/strong> – besides of course for security hunting<\/strong>.<\/p>\n\n\n\n

As outlined in the last section of this intro\/overview, I have prepared a series of blog posts to master Azure logging<\/strong> in depth<\/strong>. I will be releasing 19 blog-posts<\/strong> as shown below.<\/p>\n\n\n\n

You will learn about Azure Log Ingestion Pipeline<\/strong>, Azure Data Collection Rules<\/strong>, Data Collection Endpoints<\/strong>, Azure LogAnalytics<\/strong> custom table (v2)<\/strong>, Azure Monitor Agent<\/strong>, Azure Monitor Private Link Scope<\/strong>, migration from v1 to v2<\/strong>, etc.<\/p>\n\n\n\n

To get you started, I have also prepared templates & scripts in my AzureLogLibrary (github)<\/a> – with focus on the new capabilities.<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

Happy reading \ud83d\ude42
<\/p>\n\n\n

\n
\n\t\t\t
\n\t\t\t

Understanding Azure logging, DCRs, DCEs, AMA & transformations in depth<\/p>\n\t\t\t

<\/span><\/div>\n\t\t<\/div>\n\t\t\t
\n\n

Understanding Azure logging capabilities in depth<\/a><\/p>\n\n\n\n

How to do data transformation using Workspace transformation for legacy upload methods<\/a> <\/p>\n\n\n\n

Understanding the fundamentals of log-collection with Azure Monitor Agent & Azure Data Collection Rules<\/a> <\/p>\n\n\n\n

Understanding Azure Data Collection Endpoint<\/a> <\/p>\n\n\n\n

Optimize Costs using Auxiliary Logs for Verbose Logging<\/a><\/p>\n\n<\/div>\n\t\t<\/div>\n\n

\n\t\t\t
\n\t\t\t

Collecting data using Azure Monitor Agent (AMA)<\/p>\n\t\t\t

<\/span><\/div>\n\t\t<\/div>\n\t\t\t
\n\n

Collecting Security events using Azure Monitor Agent<\/a><\/p>\n\n\n\n

Collecting System & Application events using Azure Monitor Agent<\/a><\/p>\n\n\n\n

Collecting Performance data using Azure Monitor Agent, VMInsights and ServiceMap<\/a><\/p>\n\n\n\n

Collecting IIS logs using Azure Monitor Agent<\/a><\/p>\n\n\n\n

Collecting text logs using Azure Monitor Agent<\/a><\/p>\n\n\n\n

Collecting Syslogs using Azure Monitor Agent<\/a><\/p>\n\n\n\n

Collecting CEF Syslogs using Azure Monitor Agent<\/a><\/p>\n\n<\/div>\n\t\t<\/div>\n<\/div>\n\n

\n
\n\t\t\t
\n\t\t\t

Transformations of data<\/p>\n\t\t\t

<\/span><\/div>\n\t\t<\/div>\n\t\t\t
\n\n

Tutorial – How to make data transformations using Data Collection Rules?<\/a><\/p>\n\n<\/div>\n\t\t<\/div>\n\n

\n\t\t\t
\n\t\t\t

Troubleshooting & Monitoring of Data Collection Rules<\/p>\n\t\t\t

<\/span><\/div>\n\t\t<\/div>\n\t\t\t
\n\n

Troubleshooting & Monitoring of Data Collection Rules<\/a><\/p>\n\n<\/div>\n\t\t<\/div>\n<\/div>\n\n\n

If you are sending data using HTTP Data Collector API<\/strong> (REST)<\/strong> today, you should continue reading, as this API will be deprecated, as part of the transition to Log ingestion API<\/strong> using Azure Data Collection Rules<\/strong>, Azure Pipeline<\/strong>, Azure LogAnalytics custom tables (v2)<\/strong>.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

As you can see from the illustrations above more components (DCR, DCR, Pipeline, Schema) are added, which also increases the complexity<\/strong>.<\/p>\n\n\n\n

I have built a Powershell module, AzLogDcrIngestPS<\/a> which will ease the steps, if you want to send any data to Azure LogAnalytics custom logs (v2)<\/strong> \u2013 using the new features of Azure Log Ingestion Pipeline<\/strong>, Azure Data Colection Rules & Log Ingestion API<\/strong>.<\/p>\n\n\n\n

You can find AzLogDcrIngestPS in Powershell Gallery<\/a><\/p>\n\n\n

\n
\n\t\t\t
\n\t\t\t

My Powershell module AzLogIngestPS for Azure Log Ingestion API<\/p>\n\t\t\t

<\/span><\/div>\n\t\t<\/div>\n\t\t\t
\n\n

AzLogDcrIngestPS \u2013 your helper to send data via Azure Pipeline, Azure Log Ingestion API & Azure Data Collection Rules into Azure LogAnalytics table<\/a><\/p>\n\n\n\n

AzLogDcrIngestPS \u2013 how to do data manipulation before sending data via Azure Pipeline, Log ingestion API & Azure Data Collection Rules into Azure LogAnalytics ?<\/a><\/p>\n\n\n\n

AzLogDcrInstPS \u2013 how to do transformation of your data using Azure Data Collection Rules into Azure LogAnalytics?<\/a><\/p>\n\n\n\n

AzLogDcrIngestPS – tips & tricks sending data via Azure Pipeline, Azure Log Ingestion API, Azure Data Collections into Azure LogAnalytics<\/a><\/p>\n\n\n\n

Introducing \u201cLog-hub\u201d \u2013 sending your data through intermediate hub into Azure Pipeline, Log Ingestion API, Azure LogAnalytics<\/a> (github)<\/p>\n\n<\/div>\n\t\t<\/div>\n<\/div>\n\n

\n
\n\t\t\t
\n\t\t\t

ClientInspector – showcase for AzLogIngestPS and Azure Log Ingestion<\/p>\n\t\t\t

<\/span><\/div>\n\t\t<\/div>\n\t\t\t
\n\n

ClientInspector \u2013 a cool showcase to demonstrate Log ingestion API, Azure Log Ingestion Pipeline, Azure Data Collection Rules and my new Powershell module AzLogDcrIngestPS<\/a><\/p>\n\n\n\n

ClientInspector-DeploymentKit – How can I get started with ClientInspector so I can play with Azure Pipeline, Azure Data Collection Rules and Azure LogAnalytics (Gibhub)?<\/a><\/p>\n\n<\/div>\n\t\t<\/div>\n<\/div>\n\n\n

Lastly I would like to thank the product teams in Seattle, USA<\/strong> and Israel<\/strong>, who are delivering rock star products every day. <\/p>\n\n\n\n

Especially, I would like to give big credits<\/strong> to a few people, who I have worked together with on building AzLogDcrIngestPS Powershell module<\/strong> and my daily work with the Azure log & viewing capabilities<\/strong>:<\/p>\n\n\n\n

Name<\/th>Role<\/th><\/tr><\/thead>
Ivan Varnitski<\/td>Program Manager – Azure Pipeline<\/td><\/tr>
Evgeny Ternovsky<\/td>Program Manager – Azure Pipeline<\/td><\/tr>
Nick Kiest<\/td>Program Manager – Azure Data Collection Rules<\/td><\/tr>
Oren Salzberg<\/td>Program Manager – Azure LogAnalytics<\/td><\/tr>
Guy Wild<\/td>Technical Writer – Azure LogAnalytics<\/td><\/tr>
John Gardner<\/td>Program Manager – Azure Workbooks<\/td><\/tr>
Shikha Jain<\/td>Program Manager – Azure Workbooks<\/td><\/tr>
Ingo Bringemeier<\/td>Principal Program Manager – Azure Monitor<\/td><\/tr>
Shayoni Seth<\/td>Program Manager – Azure Monitor Agent<\/td><\/tr>
Jeff Wolford<\/td>Program Manager – Azure Monitor Agent<\/td><\/tr>
Xema Pathak<\/td>Program Manager – Azure VMInsight (integration to Azure Monitor Agent)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\"\"<\/figure>\n","protected":false},"excerpt":{"rendered":"

I am really passioned about the logging capabilities in M365 Defender and Azure with the power to bring data back from clients, servers, cloud and … <\/p>\n

Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":2155,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"ngg_post_thumbnail":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[140,55,59,129,131,61,128,54,143,141,57,28,139,136,138,137,142,133,134,146,60,132,58,145,135,144],"tags":[],"class_list":["post-1687","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ama","category-azure","category-azure-arc","category-azure-data-collection-rules","category-azure-data-ingestion-pipeline","category-azure-extensions","category-azure-log-ingestion-api","category-azure-loganalytics","category-azure-logging","category-azure-monitor-agent","category-azure-security","category-community","category-data-collection-endpoint","category-data-transformation","category-dce","category-dcr","category-logging","category-microsoft-security","category-mvpbuzz","category-performance-collection","category-scripting","category-security","category-sentinel","category-servicemap","category-transformation","category-vminsight","infinite-scroll-item","resize-featured-image"],"featured_image_src":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/53A1655_fritlagt_udenskygge.png","author_info":{"display_name":"Morten Knudsen","author_link":"https:\/\/mortenknudsen.net\/?author=1"},"jetpack_featured_media_url":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/53A1655_fritlagt_udenskygge.png","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1687","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1687"}],"version-history":[{"count":63,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1687\/revisions"}],"predecessor-version":[{"id":3342,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1687\/revisions\/3342"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/media\/2155"}],"wp:attachment":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1687"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1687"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1687"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}