Query Languages

Common Security Events, how to analyze them, and the tools to do so

Triaging and investigating security events are the bread and butter of any SOC analyst. Before we can investigate we must first have two things: A common terminology to define attacks and security related activity, and searches to direct us to that activity.

Event search languages and rules

Weather it is an open source tool like Elastic Stack, or a proprietary tool like Splunk, every data mining or security event generating tool, has its own language it works in. Specifically for SIEMs, EDRs, and Data mining platforms, they require an extensive and flexible language to search for data with a multitude of conditions that a user might have. Writing effective searches is an artform, and it is highly encouraged that you do tons of research and testing before deployment. Blue team blog has a fantastic Use Case writing guide as well as some fantastic use cases available for free.

Sigma

The syntax and format of all of the available languages is vast and complex. Especially so, when you have to convert search parameters from one tool to another. In comes Sigma.

"Sigma is a generic and open signature format that allows you to describe relevant log events in a straightforward manner. The rule format is very flexible, easy to write and applicable to any type of log file. The main purpose of this project is to provide a structured form in which researchers or analysts can describe their once developed detection methods and make them shareable with others.Sigma is for log files what Snort is for network traffic and YARA is for files." - Sigma Github Notes

Sigma is a fantastic tool that decouples rule logic from vendor terminology. Sigma is stored in easy to ready YAML format and is compatible with the MISP intel tool.

Remember two things: First, Sigma queries may not be perfect, but they should get you 90-95% the way towards what you are looking for. Be prepared to tweak! Second, online converters might not be able to translate from one platform easily. Example: LogRhythm -> Splunk. Sometimes you need to look at various Github repositories to get the code to convert to Sigma, then on to the platform of your choice. Example: LogRhythm -> Sigma -> Splunk.

• GitHub - Neo23x0/sigma: Generic Signature Format for SIEM Systems

• GitHub - socprime/SigmaUI: SIGMA UI is a free open-source application based on the Elastic stack and Sigma Converter (sigmac)

• How to Write Sigma Rules - Nextron Systems

• GitHub - LogRhythm-Labs/Sigma: Convert Sigma rules to LogRhythm searches

• https://techcommunity.microsoft.com/t5/Azure-Sentinel/Importing-Sigma-Rules-to-Azure-Sentinel/ba-p/657097

• sigmaio - simple webapp for converting sigma rules into siem queries using the pySigma library

  • https://sigmaio.herokuapp.com/

• WithSecureLabs/chainsaw - Chainsaw provides a powerful ‘first-response’ capability to quickly identify threats within Windows forensic artefacts such as Event Logs and MFTs. Chainsaw offers a generic and fast method of searching through event logs for keywords, and by identifying threats using built-in support for Sigma detection rules, and via custom Chainsaw detection rules.

• Yamato-Security/hayabusa - Hayabusa (隼) is a sigma-based threat hunting and fast forensics timeline generator for Windows event logs.

To make Sigma even easier to use, there are tools like Uncoder.io that can easily translate the rule syntax from one platform to another, free of charge. As stated above, if you cannot do direct translations from one platform to another, try changing the source search to Sigma first, then on to the platform of your choice.

• Online translator for SIEM saved searches, filters, queries and Sigma rules - Uncoder.IO

Lucene (ElasticSearch)

Apache's search language that is used in many technologies including Elastic Stack and Palo Alto's XSOAR

• https://logz.io/blog/elasticsearch-queries/

• https://lucene.apache.org/core/2_9_4/queryparsersyntax.html

• KQL: Kibana search language based on Lucene. - https://www.elastic.co/guide/en/kibana/master/kuery-query.html

• Elasticsearch: The Definitive Guide (fork it on GH)

• Threat Hunting in Elastic Stack: Lucene - pg. 212

Solr - Solr is the popular, blazing-fast, open source enterprise search platform built on Apache Lucene

• https://github.com/hectorcorrea/solr-for-newbies

KQL - Kibana Query Language, the default query language of the Kibana seach and alerting utility within Elastic Stack.

• Threat Hunting in Elastic Stack: KQL - pg. 216

EQL - Elastic Query Language, an advanced query language developed by Elastic for use in thier Security App.

• Threat Hunting in Elastic Stack: EQL - pg. 220

Splunk's SPL: Search Processing Language

• https://docs.splunk.com/Documentation/Splunk/8.2.1/SearchTutorial/Usethesearchlanguage

• https://gosplunk.com/

• https://wiki.splunk.com/images/2/2b/Cheatsheet.pdf

• https://www.splunk.com/pdfs/solution-guides/splunk-quick-reference-guide.pdf

• https://www.splunk.com/pdfs/solution-guides/splunk-dashboards-quick-reference-guide.pdf

• https://docs.splunk.com/Documentation/Splunk/8.1.0/SearchReference/ListOfSearchCommands

• Operator Handbook: Splunk - pg. 277

Graylog

• Graylog's query language is very close to Lucene. Watch for syntax errors.

  • https://docs.graylog.org/en/3.3/pages/searching/query_language.html

Windows O365

• https://docs.microsoft.com/en-us/microsoft-365/security/defender/advanced-hunting-query-language?view=o365-worldwide

Windows Defender ATP

• https://docs.microsoft.com/en-us/windows/security/threat-protection/

• https://github.com/alexverboon/WindowsDefenderATP-Hunting-Queries

• Operator Handbook: Windows Defender ATP - pg. 417