{"id":450,"date":"2022-12-29T23:47:36","date_gmt":"2022-12-29T22:47:36","guid":{"rendered":"https:\/\/mortenknudsen.net\/?p=450"},"modified":"2023-01-02T09:51:30","modified_gmt":"2023-01-02T08:51:30","slug":"example-of-sentinel-cost-optimization-with-43-savings-how-to-exclude-syslog-log-events-from-banned-ips-using-abuseipdb-service-with-integration-to-firewalls","status":"publish","type":"post","link":"https:\/\/mortenknudsen.net\/?p=450","title":{"rendered":"Real example with 43% cost savings on Sentinel log-costs: How to exclude Syslog log-events from banned IPs using AbuseIPDB-service with integration to firewalls"},"content":{"rendered":"\n

This is a real-life example of how I helped reduce the log-cost by 43%<\/strong> for LogAnalytics & Sentinel<\/strong> combined for a customer – by filtering banned IPs out of Syslog-data using API from AbuseIPDB<\/strong>. Banned IPs are for example port-scanners on the internet.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Some time ago, I analyzed the Syslog Traffic events<\/strong> going into a customers CommonSecurityLog<\/strong> table<\/strong> in Azure LogAnalytics.<\/p>\n\n\n\n

I discovered that ~ 50% of the traffic<\/strong> was coming from port-scanners on the internet <\/strong>scanning the public IPs<\/strong> of the main firewall<\/strong> and branch office firewalls. <\/p>\n\n\n\n

\"\"
Sample of data after enabling the exemptions, you can read about in this blog<\/figcaption><\/figure>\n\n\n\n

Of course my customer drops this traffic<\/strong>, but my customer also wanted to drop the syslog entry of the events caused by these banned IP addresses to save money<\/strong> (removing the mentioned 50%). They use Azure LogAnalytics for logging and Microsoft Sentinel as SIEM.<\/p>\n\n\n\n

From a security hunting perspective, some of you would say, that you want this information kept in your logs, both for detections & investigations and compliance reason – whereas others would argue that this is an accepted risk exemption<\/strong>. Choose your security-level yourself \ud83d\ude42<\/p>\n\n\n\n

Design Considerations<\/h5>\n\n\n\n

When designing your Azure LogAnalytics environment, consider to use commitment tier or dedicated cluster; Basic logs and Archive logs. I have also covered the new data transformation in another blog-post<\/a>. More cost optimization tips are covered here<\/a><\/p>\n\n\n\n

Configure pricing tier or dedicated cluster for your Log Analytics workspaces<\/td>By default, Log Analytics workspaces will use pay-as-you-go pricing with no minimum data volume. If you collect enough amount of data, you can significantly decrease your cost by using a commitment tier<\/a> or dedicated cluster<\/a>, which allows you to commit to a daily minimum of data collected in exchange for a lower rate.

See Azure Monitor Logs cost calculations and options<\/a> for details on commitment tiers and guidance on determining which is most appropriate for your level of usage. See Usage and estimated costs<\/a> to view estimated costs for your usage at different pricing tiers.<\/td><\/tr>
Configure tables used for debugging, troubleshooting, and auditing as Basic Logs<\/td>Tables in a Log Analytics workspace configured for Basic Logs<\/a> have a lower ingestion cost in exchange for limited features and a charge for log queries. If you query these tables infrequently, this query cost can be more than offset by the reduced ingestion cost.<\/td><\/tr>
Configure data retention and archiving<\/td>There is a charge for retaining data in a Log Analytics workspace beyond the default of 30 days (90 days in Sentinel if enabled on the workspace). If you need to retain data for compliance reasons or for occasional investigation or analysis of historical data, configure Archived Logs<\/a>, which allows you to retain data for up to seven years at a reduced cost.

See Configure data retention and archive policies in Azure Monitor Logs<\/a> for details on how to configure your workspace and how to work with archived data.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Is there another design option? YES<\/strong> with ADX – <\/a>Azure Data Explorer<\/strong>. See teaser at the very end of this blog-post. I will cover this in a later blog-post.<\/p>\n\n\n\n

Implementation strategy we chose to skip Syslog events from Banned IPs<\/h2>\n\n\n\n

Going back to my customer-case, when we were looking into our options, we found, that some of the firewall vendors provided malicious IP-list capabilities, but our investigations concluded that they were not updated as frequent as we wanted (at least at the time of implementation), so we decided to come up with the solution outlined below:<\/p>\n\n\n\n