{"id":1436,"date":"2023-04-02T19:06:05","date_gmt":"2023-04-02T18:06:05","guid":{"rendered":"https:\/\/mortenknudsen.net\/?p=1436"},"modified":"2023-04-03T11:50:04","modified_gmt":"2023-04-03T10:50:04","slug":"how-to-do-data-transformation-using-workspace-transformation-for-legacy-upload-methods","status":"publish","type":"post","link":"https:\/\/mortenknudsen.net\/?p=1436","title":{"rendered":"How to do data transformation using Workspace transformation for legacy upload methods"},"content":{"rendered":"\n

This blog will demonstrate how you can do workspace transformation to support legacy data transformation<\/strong> where data is being uploaded by legacy methods.<\/p>\n\n\n\n

You cannot do a workspace transformation, if the sources sending data is coming from Azure Monitor Agent or Log Ingestion API – only legacy agent\/solution (MMA)<\/strong> and legacy HTTP Data Collector API<\/strong><\/p>\n\n\n\n

This blog-post is part of a series of blog posts to master Azure logging in depth (overview)<\/a>. <\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

<\/p>\n\n\n\n

To get you started, you can find ARM-templates & scripts in my AzureLogLibrary (github)<\/a>. Details will be covered in the articles.<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n


<\/p>\n\n\n\n

Quick Links<\/h1>\n\n\n\n

Data transformation<\/a>
How to setup in GUI ?<\/a>
Real-life example of workspace transformation<\/a>
More information (Microsoft)<\/a>

<\/p>\n\n\n\n

Data Transformation<\/h1>\n\n\n\n

If you want to read about the data transformation capabilities, please read more here<\/a><\/p>\n\n\n\n

Currently, Microsoft supports doing transformation using 4 methods:<\/p>\n\n\n\n

Collection source<\/th>How<\/th>Purpose \/ limitatations<\/th><\/tr><\/thead>
XPath<\/td>XML syntax in DCR<\/td>XPath implementation uses v1, which is very limited in functionality.<\/td><\/tr>
MMA-agent \/ Solutions (legacy):
Performance
Eventlog
Syslog

API (legacy):
HTTP Data Collector API<\/td>		Workspace transformation DCR<\/td>Only one transformation per table<\/td><\/tr>
All sources sending in using AMA<\/td>AMA transformation DCR<\/td>All DCRs do unions, so be aware of double data. Governance is important<\/td><\/tr>
REST API using Log ingestion API<\/td>Log Ingestion transformation DCR<\/td><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

<\/p>\n\n\n\n

\n

You cannot do a workspace transformation, if the sources sending data is coming from Azure Monitor Agent or Log Ingestion API – only legacy agent\/solution (MMA)<\/strong> and legacy HTTP Data Collector API<\/strong><\/p>\n<\/blockquote>\n\n\n\n

<\/p>\n\n\n\n

How to setup in GUI ?<\/h1>\n\n\n\n

You can setup a workspace transformation in the GUI on the table. <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

You can find the transformation inside the Data Collection Rule. It is important to choose the newest API version (2022-06-01) in the right corner.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n


<\/p>\n\n\n\n

Real-life example of workspace transformation<\/h1>\n\n\n\n

Here is an example where I did a workspace transformation removing Syslog events<\/strong> with specific patterns. As you can see, the results were a significant decrease. <\/p>\n\n\n\n

Of course there is a trade-off, which must be considered. There are always 2 sides of the coin – will I miss the data at some point !<\/p>\n\n\n\n

source\n| where (DeviceVendor !contains \"sonicwall\") or ((DeviceVendor contains \"sonicwall\") and (Activity contains \"connection opened\" or Activity contains \"connection closed\") and (Protocol != \"udp\/dns\"))<\/code><\/pre>\n\n\n\n

<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n
More information (Microsoft)<\/h1>\n\n\n\n
https:\/\/learn.microsoft.com\/en-us\/azure\/azure-monitor\/logs\/tutorial-workspace-transformations-portal<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
This blog will demonstrate how you can do workspace transformation to support legacy data transformation where data is being uploaded … <\/p>\n
Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":1783,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"ngg_post_thumbnail":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[140,55,59,129,131,61,128,54,143,141,57,28,139,136,138,137,142,133,134,146,60,132,58,145,135,144],"tags":[],"class_list":["post-1436","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ama","category-azure","category-azure-arc","category-azure-data-collection-rules","category-azure-data-ingestion-pipeline","category-azure-extensions","category-azure-log-ingestion-api","category-azure-loganalytics","category-azure-logging","category-azure-monitor-agent","category-azure-security","category-community","category-data-collection-endpoint","category-data-transformation","category-dce","category-dcr","category-logging","category-microsoft-security","category-mvpbuzz","category-performance-collection","category-scripting","category-security","category-sentinel","category-servicemap","category-transformation","category-vminsight","infinite-scroll-item","resize-featured-image"],"featured_image_src":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/workspace-transformation-example.png","author_info":{"display_name":"Morten Knudsen","author_link":"https:\/\/mortenknudsen.net\/?author=1"},"jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"jetpack_featured_media_url":"https:\/\/mortenknudsen.net\/wp-content\/uploads\/2023\/04\/workspace-transformation-example.png","_links":{"self":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1436","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1436"}],"version-history":[{"count":24,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1436\/revisions"}],"predecessor-version":[{"id":2141,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/posts\/1436\/revisions\/2141"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=\/wp\/v2\/media\/1783"}],"wp:attachment":[{"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1436"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1436"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mortenknudsen.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1436"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}