Demystifying Log Collection in Azure: Navigating Windows and Linux Server Logging for Microsoft Sentinel
I'm crafting this blog post to shed light on the target tables in Log Analytics, specifically for non-Azure server logs. Just a heads-up: this article won't cover the setup process for log ingestion. However, I'll make sure to include pertinent links throughout.
Many of my customers, and initially myself, have found the target tables in Log Analytics to be a bit puzzling, especially when setting up log collection in support of Microsoft Sentinel. In this blog, I aim to clear up any confusion and offer a few handy recommendations.
In this discussion, I won't be delving into the Microsoft Monitoring Agent (MMA), as it's set to be deprecated in August 2024. Instead, I'll concentrate on Windows and Linux servers that are either equipped with the Azure Monitor Agent (AMA) or don't use any agent. Let's start by taking a closer look at Linux servers.
Linux
There are two main approaches to retrieving logs from a Linux server. The first involves forwarding the logs to a Syslog server, which then sends them through the Azure Monitor Agent (AMA) to the Log Analytics Workspace supporting Microsoft Sentinel. The second approach is to install the AMA directly on the server, enabling it to send logs straight to the Log Analytics Workspace that integrates with Microsoft Sentinel.
If you prefer not to install the Azure ARC agent and its AMA extension on all your servers, or if you have devices capable of exporting Syslog, opting for a Syslog forwarder is a good choice. This forwarder operates on the standard TCP/UDP port 514 for syslog. It then utilizes Azure ARC and AMA to transmit the events to the Log Analytics Workspace over the secure TCP port 443. This is done using the "Common Event Format (CEF) via AMA" connector, a native feature of Microsoft Sentinel. The message format you send determines its destination table: CEF-formatted messages land in the CommonSecurityLog table, while Syslog or unrecognized formats end up in the Syslog table.
Suppose you install the Azure ARC agent and AMA extension directly onto a system. In that case, you can set up Data Collection Rules directly from Azure Monitor or the Microsoft Sentinel "Syslog Connector." Their format determines the destination table for your messages in the Log Analytics Workspace: messages in Common Event Format (CEF) are directed to the CommonSecurityLog table. In contrast, messages in Syslog or unrecognized formats are routed to the Syslog table.
Windows
Now, let's shift our focus to Windows Events from Windows devices. The method you choose for collecting these events determines where they are stored. If you use a Windows Event Forwarder, the events will be sent to the WindowsEvent table. On the other hand, if the events are collected through a Data Collection Rule (DCR) created via Azure Monitor, they'll end up in the Event table. Lastly, if the DCR is established using the Sentinel connector, the events will be directed to the SecurityEvent table.
Just as with the Syslog forwarder scenario, there might be situations where installing the Azure ARC agent with the AMA extension on every Windows device isn't ideal. In these instances, Windows Event Forwarding is a viable alternative. This feature, integrated into modern Windows operating systems, enables your systems to send Windows Events to a centralized collector. This collector then forwards the events to the Log Analytics Workspace, utilizing the Azure ARC agent and the AMA extension. You'll use the "Windows Forwarded Event" connector in Microsoft Sentinel to set up this data collection. All data transmissions are securely handled. When you use Windows Event Forwarding, the collected Windows Events are stored in the table.
If your plan involves collecting Windows Events directly from a device using the Azure ARC agent with the AMA extension, then employing the "Windows Security Events via AMA" connector in Microsoft Sentinel is recommended. In a later discussion, I'll delve into the limitations of crafting your own Data Collection Rules (DCRs) via Azure Monitor. Opting for this method creates a Sentinel-created special DCR, which efficiently forwards Windows Events to the SecurityEvent table.
If your strategy includes gathering Windows Events directly from a device using the Azure ARC agent and the AMA extension and setting up the Data Collection Rules (DCRs) directly via Azure Monitor, your Windows Events will be stored in the Event table. You can still construct analytics with Microsoft Sentinel using this table. However, there's a catch if you plan to utilize the User and Entity Behavior Analytics (UEBA) features of Microsoft Sentinel. Currently, UEBA examines all the other tables I've mentioned earlier but does not analyze the Event table. It's my hope that this limitation will be addressed in future updates.
Following the Data Collection Rules Creation Impacting Sentinel UEBA ML Model blog, you can alter this behavior. Additionally, you can streamline the process by installing the "Data Collection Rule Toolkit" workbook available in the Content Hub of Microsoft Sentinel. This allows you to modify the Data Collection Rule (DCR) directly, without using PowerShell, by implementing the JSON changes described in the blog. This approach offers a more user-friendly and efficient way to adjust your data collection setup.
The complete picture:
