Real example with 43% cost savings on Sentinel log-costs: How to exclude Syslog log-events from banned IPs using AbuseIPDB-service with integration to firewalls

This is a real-life example of how I helped reduce the log-cost by 43% for LogAnalytics & Sentinel combined for a customer – by filtering banned IPs out of Syslog-data using API from AbuseIPDB. Banned IPs are for example port-scanners on the internet.

Some time ago, I analyzed the Syslog Traffic events going into a customers CommonSecurityLog table in Azure LogAnalytics.

I discovered that ~ 50% of the traffic was coming from port-scanners on the internet scanning the public IPs of the main firewall and branch office firewalls.

Of course my customer drops this traffic, but my customer also wanted to drop the syslog entry of the events caused by these banned IP addresses to save money (removing the mentioned 50%). They use Azure LogAnalytics for logging and Microsoft Sentinel as SIEM.

From a security hunting perspective, some of you would say, that you want this information kept in your logs, both for detections & investigations and compliance reason – whereas others would argue that this is an accepted risk exemption. Choose your security-level yourself 🙂

Design Considerations

When designing your Azure LogAnalytics environment, consider to use commitment tier or dedicated cluster; Basic logs and Archive logs. I have also covered the new data transformation in another blog-post. More cost optimization tips are covered here

Configure pricing tier or dedicated cluster for your Log Analytics workspaces	By default, Log Analytics workspaces will use pay-as-you-go pricing with no minimum data volume. If you collect enough amount of data, you can significantly decrease your cost by using a commitment tier or dedicated cluster, which allows you to commit to a daily minimum of data collected in exchange for a lower rate. See Azure Monitor Logs cost calculations and options for details on commitment tiers and guidance on determining which is most appropriate for your level of usage. See Usage and estimated costs to view estimated costs for your usage at different pricing tiers.
Configure tables used for debugging, troubleshooting, and auditing as Basic Logs	Tables in a Log Analytics workspace configured for Basic Logs have a lower ingestion cost in exchange for limited features and a charge for log queries. If you query these tables infrequently, this query cost can be more than offset by the reduced ingestion cost.
Configure data retention and archiving	There is a charge for retaining data in a Log Analytics workspace beyond the default of 30 days (90 days in Sentinel if enabled on the workspace). If you need to retain data for compliance reasons or for occasional investigation or analysis of historical data, configure Archived Logs, which allows you to retain data for up to seven years at a reduced cost. See Configure data retention and archive policies in Azure Monitor Logs for details on how to configure your workspace and how to work with archived data.

Is there another design option? YES with ADX – Azure Data Explorer. See teaser at the very end of this blog-post. I will cover this in a later blog-post.

Implementation strategy we chose to skip Syslog events from Banned IPs

Going back to my customer-case, when we were looking into our options, we found, that some of the firewall vendors provided malicious IP-list capabilities, but our investigations concluded that they were not updated as frequent as we wanted (at least at the time of implementation), so we decided to come up with the solution outlined below:

• We wanted to subscribe to an external well-known web-service, that could deliver updated insight (real-time) if an IP address was considered as malicious / abuse. It should support ConfidenceScore
  • We decided to go with AbuseIPDB, where we could buy an API-access and there is a fantastic community of companies sending abuse-data into their backend.
  • NOTE: AbuseIPDB are not paying me anything to write this blog. Maybe there are similar services “out-there”, but that was the service we chose. And I have found it to be very stable and useful.
• We wanted the logic for ‘drop-traffic’ & ‘drop-syslog-event’ to be done on the firewall close to the entry-point, so we didn’t have to transform data in the LogAnalytics backend in a pipeline.
• We wanted to build a script (script #1 – provided below), that could extract the list of IP addresses from CommonSecurityLog and run a API-call against AbuseIPDB to check if an ip address was reported banned – or it was clean
  • The logic should build a single TXT-file with a list of banned ip-addresses
  • Lastly the script should upload the file to an Azure App Service, which was only accessible from the firewalls.
• Every 5 min (or other frequency), the firewall should import the file and update its cache of banned IPs
  • When traffic was seen from any of these ip addresses, a rule would drop the traffic and drop the syslog event entry, so no syslog was sent to LogAnalytics
• Every 24 hours, another script (script #2 – provided below) should re-analyze the banned list of IPs to remove any IPs, which had been whitelisted again during the last 24 hours on AbuseIPDB.

Results – Customer gained a 43% drop of their log & Sentinel-costs !!

After 48 hours, the log-cost for this customer dropped with 43% for Azure LogAnalytics and Sentinel combined, due to this optimization of Syslog-data.

Scripts

You can find the 2 scripts using the github link

Each of the scripts are being detailed later in this blog.

Maybe there are more built-in ways to handle this today, compared to our choices approx 24 months ago. But the solution is running perfectly today – and it is a great example of how you can optimize your log-costs by thinking out of the box 🙂

Script #1 | AbuseIPDB-Check.ps1 |Runs every 1 hour

Script #2 – CheckAbuseWhitelistedIPAddresses.ps1 |Runs every 24 hours

Is there an alternative option? YES (ADX – Azure Data Explorer)

An alternative option is to use a short retention of 1-3 month of the data in Sentinel/LogAnalytics for security hunting – and then send data into a Azure Data Explorer cluster for longer retention and compliance reasons.

I will cover this in another later blog-post.