What is Splunk?
Splunk is a software platform to search, analyze and visualize the machine-generated data gathered from the websites, applications, sensors, devices, which make up your IT infrastructure and business.
To put this into perspective imagine trying to identify suspicious network behavior or perhaps performance issues on a host with nothing but but raw logs to review. Raw traffic and event log examples below.
Splunk takes the raw logs and data, processes and presents a visual format for the end user with pre-built modules, automation and search queries.
Log parsing becomes very efficient in Splunk partly thanks to the selection and interest fields. This are predefined search queries. Enabling you to parse thousands of logs in seconds for particular information.
Splunk offers a wide range of features and capabilities allowing for greater insight and awareness into an environment. SMTP parsing is one of those capabilities, allowing for custom email search queries targeting metadata. This feature can be added by downloading the StoQ addon. There are many other add on's to improve and tailor the use of Splunk.
Threat Hunting with Splunk- What to look for?
Where do we start when malware is suspected in the environment? This can be a challenging question and below we will review a few starting points, addressing the what and why, as well as the required data and queries.
Suspicious File names
• What: Suspicious file names,
• Why: Attackers may attempt to launch programs and commands through powershell or executables. To avoid detection attackers may encode, obfuscate, bury or add double and fake file extensions.
• Data needed: Endpoint system logs.
• How to query: host=MyPc | rare limit=20 file_name
Uncommon Executable's
• What: Executable programs or scripts, especially anything ending in .exe or ps1.
• Why: The attacker has to install and run malware to establish deep control and persistence.
• Data needed: Log of running processes, and scheduled tasks or cron jobs.
• How to query: host=MyPc file_name="*" app="*"| rare limit=20 app
OR
Searching for identical executable names with different hashes.
sourcetype=”XmlWinEventLog:Microsoft-Windows-Sysmon/Operational” Image=*svchost.exe
| dedup Computer
| eval TIME=strftime(_time,”%Y-%m-%d %H:%M”) | stats first(TIME) count by Image, Hashes
Windows Start up scripts/processes
• What: Modification of HKLM\SOFTWARE\Microsoft\Windows\Current\Version\Run
• Why: Malware will seek to establish persistence, which largely relies on executing scripts on startup.
• Data needed: Active monitoring of Microsoft Windows registry with Splunk or RegMon piping data into Splunk.
• How to query: index="" sourcetype=WinRegistry key_path="HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run *
Covering your tracks
• What: Signs of event logs being deleted.
• Why: The attacker may aim to delete some or all evidence relating to their activities. Often premade scripts and tools used by attacks will delete much more than is necessary.
• Data needed: Microsoft Windows event logs
• How to query: host=MyPC EventCode=1102 OR 517
Suspicious outbound DNS traffic
• What: Large volumes of DNS requests from an internal host, or patterns of same sized requests.
• Why: Possible threat communications, ex-filtration of data, relaying instructions or downloading files.
• Data needed: DNS logs
• How to query: tag=dns message_type="Query" | timechart span=* limit=10 usenull=f useother=f count AS Requests by src
OR
We can look for signs of data exifiltration through packet size and volume
tag=dns message_type="QUERY" | mvexpand query | eval queryLength=len(query) | stats count by queryLength, src | sort -queryLength, count | table src queryLength count
Phoning Home - AKA- Beaconing
• What: Traffic to the same URL at the same interval every day or blacklisted domains.
• Why: Malware trying to establish communication with a command and control server.
• Data needed: Web proxy logs or firewall logs.
• How to query: index="MyData"
| convert mktime(_time) as epoch
| sort 0 uri_host,client_ip,epoch
| delta epoch as epoch_delta | search epoch_delta>0 epoch_delta<30
| chart count over epoch_delta by uri_host
OR
index=”*” type=”dns”
| fields timestamp,dnsdomainname,srcip
| streamstats last(timestamp) as last, first(timestamp) as first by dnsdomainname
| eval gap=(last – first)/1000
| stats avg(gap) as “avgbeacontime”,var(gap) as “variancebeacon” by dnsdomainname
Command and Control Server
• What: Traffic to unknown site not listed by a reputation service or filter.
• Why: Attackers will use new or low traffic domains that have not been evaluated by reputation engines
• Data needed: Web proxy logs or firewall logs with the reputation tag.
• How to query: source=proxy sc_filter_category=None OR sc_filter_ category=unknown| stats count(clientip) by s_ hostname, clientip
Malware Delivery and Installation
• What: Rapid traffic after the downloading of an executable file (PDF, Java, .exe, etc.)
• Why: Signs of exploitation, and downloading additional files/
• Data needed: Web proxy or firewall data that includes complete URL or file names
• How to query: source=proxy [search file=*.pdf OR file=*.exe | dedup clientip | table clientip] | transaction maxspan=60s maxpause=5s clientip | eval Length=len(_raw) | sort –Length
Lets start hunting for threats!
I will be using Splunk as I work my way through the Sans Kringle Con Holiday hack challenge. A scenario involving one of Santa's workshop computers which has been hit with malware. We are of course tasked with investigating, and finding all relevant information.
Sensitive files
A malware author will often attempt to download more files upon successfully deploying initial malware. During this investigation it will be valuable to find out if there was a transfer of data, a downloaded file, or any suspicious file names around the time of attack. Files such as .dll's or .ps1 scripts can be a dead give away.
Lets search for top 20 rare file names using the following string.
index=main | rare limit=20 file_name
Here we are seeing that powershell.exe has been launched and the following activity is showing that powershell is attempting to obfuscate its commands.
Lets review the Powershell command flags in this instance.
-noP NoProfile - Prevents PowerShell from loading profile scripts, which get executed on launch, so as to avoid potentially unwanted commands or settings.
-sta Single threaded apartment - Uses single-threaded apartment This parameter was almost exclusively used in PowerShell Empire.
-w WindowStyle Hidden - Used to prevent PowerShell from displaying a window when it executes code.
-enc Encode - Bypasses the default PowerShell script execution policy.
This is a clear sign of malicious activity as no employee/user should be casually launching obfuscated powershell commands. We find that the malicious code is run by a parent process WmiPrvSE.exe
Anytime we see powershell.exe with a WmiPrvSE.exe parent process, we noticed a log right before it.
When we expand the event, we see that a WMI module is being loaded before the powershell.exe is suspicious.
We search for the process ID in hex, and we find the malicious file below in the parent process. WINWORD.exe has called "Century Holiday Cheer assignment", that created a child process to launch powershell.exe -noP -sta -w -enc.
Where did the malicious file come from?
Utilizing the splunk add on stoQ we can parse through email logs. We filter all logs to have the subject line of the malicious file name. "Century Holiday Cheer Assignment", and our results accordingly.
The query to parse email subject lines below.
"results{}.workers.smtp.subject"="Holiday Cheer Assignment Submission"
These all look like legitimate emails regarding an assignment that several people were working on, with professor Carl Banas.
At first glance it appears it was sent by bradly buttercups, but when you look closely.......typical phishing tactics, swap the i and l.
Command and Control
Malware often attempts to make a call back to notify the attacker of installation, or to request further instructions. This means we want to review the network traffic for all destinations.
We will first add the Destination Hostname field to our table and then search for all network activity through Windows event code 3.
index=main sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational powershell EventCode=3
This is suspicious domain, and we make note of this as we see many requests to this address.
Lest paint a picture here.
An email was opened on host SWEETUMS by Carl Banas, in particular from an imposter claiming to be Bradly buttercups. Carl Banas was expecting email attachments regarding a holiday cheer assignment, and opened the malicious attachment which made a successful callback to 144.202.46.214.vultr.com.