{"id":575,"date":"2023-01-12T01:47:00","date_gmt":"2023-01-12T00:47:00","guid":{"rendered":"https:\/\/mortenknudsen.net\/?p=575"},"modified":"2023-01-15T11:09:06","modified_gmt":"2023-01-15T10:09:06","slug":"how-to-save-by-storing-your-syslog-and-defender-for-endpoint-long-term-logs-in-azure-data-explorer-cluster-using-azure-data-factory-and-azure-storage-account-export-while-keeping-kusto-q","status":"publish","type":"post","link":"https:\/\/mortenknudsen.net\/?p=575","title":{"rendered":"How to save $$$ by storing your Syslog and Defender for Endpoint long-term logs in Azure Data Explorer cluster using Azure Data Factory and Azure Storage Account export \u2013 while keeping Kusto query functionalities ?"},"content":{"rendered":"\n
\n

This blog is about keeping long-term Sentinel logs<\/strong>, giving you insight to the options today – with great opportunities to save money \ud83d\ude42<\/p>\n\n\n\n

To be honest, I can get confused with the limitations and feature-set, as there are coming many great changes rapidly. Therefore I have prepared an detailed overview<\/strong> with 2 tables covering Decision #1 Export vs. Keep<\/strong> and Decision 2: Choosing our long-term log-storage solution<\/strong><\/p>\n\n\n\n

I covered Azure LogAnalytics basic logs and archive logs in a previous blog-post<\/a> – so I have decided to cover the solution, Azure Data Explorer<\/strong>, in this blog. <\/p>\n\n\n\n

In many cases, this solution can be a great option, where you can save lots of money. But on the flipside, it is also more complex. Hopefully this guide can get you up and running, so you can try it out. I will cover adding Syslog<\/strong> data, but it can also be used for e.g. Defender for Endpoint data (and more).<\/p>\n\n\n\n

Azure Data Explorer is an alternative to consumption-based log retention<\/strong>, where you pay a fixed price for the infrastructure for long-term log-storage based<\/strong> on Azure Data Explorer cluster<\/strong> \u2013 and then use Azure Data Factory<\/strong> to transform your data. <\/p>\nIf you want to get a tutorial on how to get a complete environment up and running using Azure Data Explorer, Azure Data Factory and export to Azure Storage Account, then keep reading, where I will try to take you through the steps.<\/cite><\/blockquote>\n\n\n\n

Decision #1 – Export vs. Keep<\/h2>\n\n\n\n

When we have to decide on a solution, we will have to decide on “should we export or keep data”. Below are the options:<\/p>\n\n\n\n

Option<\/td>Export<\/td>Export<\/td>Export<\/td>Export<\/td>Keep<\/td>Keep<\/td><\/tr>
Method<\/td>Continuously Export<\/td>Continuously Export<\/td>Ad-hoc export \u2013 Powershell<\/a><\/td>Schedule export via Query
(LogicApp, Function, Data Factory<\/td>		LogAnalytics Retention<\/td>LogAnalytics Archive<\/td><\/tr>
Target<\/td>EventHub<\/td>Storage Account<\/td><\/td><\/td><\/td><\/td><\/tr>
Format<\/td>JSON<\/td>JSON<\/td>Any format<\/td>Any format<\/td><\/td><\/td><\/tr>
Limitations<\/td>Link<\/a><\/td>Link<\/a><\/td><\/td><\/td><\/td><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

<\/p>\n\n\n\n

Decision #2 – Choosing our long-term log-storage<\/strong> solution<\/h2>\n\n\n\n

The big question is what solution to choose for our long-term logs, and here the complexity starts, where we have to take many factors into consideration: platform-knowledge<\/strong> in-house<\/strong>, limitations<\/strong>, time2enable<\/strong>, management,<\/strong> performance<\/strong>, monthly cost, scalability, security<\/strong>, etc. <\/p>\n\n\n\n

Cost-factors<\/h5>\n\n\n\n

As mentioned above, many factors must be taken into account when you look at costs. Cost-differences cannot be concluded without asking some important questions. Here are 4 samples of these questions:<\/p>\n\n\n\n