Day-10 Elasticsearch Ingest Data

Posted Sep 10, 2024

By Cyberpengunine__

3 min read

To view data in Elasticsearch, we will need to ingest the data using integration. By ingesting the data, we will be able to search through all the logs obtained from Windows Defender and Sysmon on Elasticsearch. Let’s begin today’s task of the MYDFIR SOC Analyst Challenge.

Log in to your Elasticsearch instance. On the homepage, you’ll find a button labeled “Add Integrations”. Click on this button to access a list of available integrations.

We are interested in Sysmon and Windows event logs, so we will search for them.
We will search Windows event logs to collect Windows logs.

Click on the option that says “Custom Windows Event Logs.” On the right-hand side, you’ll see the button to add the integration. Click on that.

To integrate mydfir-sysmon-winlog1, provide the requested details. For the channel name, go to Event Viewer on your Windows server instance and search for sysmon. Right-click on the “Operational” under sysmon folder, select Properties, and in the General tab, locate the Full Name option. The value in the Full Name field is your channel name.

Then leave everything as default.
For “Where to add this integration?”, we’ll add the existing host MYDFIR-Windows-Policy, then click on save and continue.

Now, the sysmon integration is added to our policy.

Let’s do the same for our Windows Defender logs. Click on “Add Integration” on the right side. Then fill in the required information. For the channel name, in Event Viewer, go to Windows Defender folder, right-click on “Operational,” select “Properties,” and copy the “Full Name” from the “General” tab.

Next, click on Advanced Options. We want to add the specific Event IDs for this challenge, so we will include 1116, 1117, and 5001 separated by commas in the Event ID field. For the location of integration, we will choose MYDFIR-Windows-Policy

Click on “Save and Continue,” then “Save and deploy changes.”
If you want to exclude the event IDs, simply place a minus sign in front of each ID.
We have added integration for both Sysmon and Windows Defender as shown in below screenshot.

To test this, go to Discover from the hamburger menu and search for sysmon. If you are unable to find sysmon or there are no search results, then let’s troubleshoot:

Go to the “Integration” option under “Management” by clicking on the hamburger icon. Search for the “event_id” field. Copy this field and paste it into the search bar on the “Discover” page. Can you see “sysmon” under “data_stream_dataset”? If not, then…move to another method for troubleshooting
Go to the “Fleet” section under “Management.” Locate the instance named “mydfir-win-" and select it. Check if any logs have been collected. If not, navigate to "Services" on the Windows Server instance and restart the "Elastic Agent" service by right-clicking on it. Are logs still not appearing?
Check the CPU status in the agent details of mydfir-win-. If it shows 'N/A,' this indicates a connectivity issue between the agent and Elasticsearch. To resolve this, go to the firewall rules we created for this challenge and add a rule to allow incoming connections to your server using the TCP protocol on port 9200, with 'Anywhere' as the source. Wait a few minutes, then refresh the winlogs in Fleet > Agents > mydfir-win-. You should now see values for CPU and Memory in the Agent Details, indicating that the connection is working. Finally, go to the Logs tab and verify that logs are visible.