Yellow - Logging and Security Architecture
Logs and the logging loggers that log them
Logging Guides
https://its.uiowa.edu/support/article/3576 - Limiting or Removing Unwanted Network Traffic at the Client
https://github.com/Neo23x0/auditd - Best Practice Auditd Configuration
https://github.com/JSCU-NL/logging-essentials - A Windows event logging and collection baseline focused on finding balance between forensic value and optimising retention.
Setting up Network Sensors - Applied Network Security Monitoring - PG. 49
Sensor Placement - Applied Network Security Monitoring - PG. 61
Logging Data Types
Before getting into the logging architecture, we need to define the various logging data types. These are the different forms of the data we use for Network Security Monitoring.
Packet Capture - A complete record of every packet traveling back and forth, with all available metadata and transaction data. This is the most detailed form of data used for monitoring, but comes with an incredible storage cost.
Session Data - Also called flow data, this is a summarized version of the target traffic that provides a significantly more flexible form for storage, while sacrificing content.
Statistical Data - Various metrics and summary of data points found in network traffic. Variable per use case.
Packet String Data (PSTR) - An intermediate form of capture between full Packet Capture and Session data. Contains specified strings from the captured traffic such as clear text protocol headers.
Log Data - Raw log files generated from devices and applications.
Alert Data - A specific event log generated from a detection device that can report anomalies or specific traffic patterns.
Logging System Components
The core components of most security monitoring programs is the Security Information and Event Management system (SIEM) and the logs that support it. The architecture behind the SIEM has many components that make up the security logging ecosystem.
Log Collector - The tools that gather up the logs.
Log Agents - Applications that can be installed on devices to collect generated logs as well as perform a huge amount of other functions including: parsing, diode support, filtering, rate control, log rotation, log buffering, encryption, priority routing, file/registry monitoring, and even alerting.
A well configured agent will perform better than an agentless solution as remote authentication is not required.
Standard SIEM Agents - Agents that are paired with their premium SIEM products have easier integration, but will often have less features than their open-source alternatives, as well as usually lack filtering capabilities at the agent.
Open Source Agents - Feature rich, premium support available, and free.
Elastic Beats - Fantastic log collection agents that specialize in certain types of logs.
Log Collection Agents focus on logs for: File integrity monitoring, Windows Events, Packet monitoring, OS metrics, Cloud infrastructure, Audit data, and device health monitoring.
Can get complicated managing so many different agents.
Threat Hunting in Elastic Stack: Filebeat - pg. 55
Threat Hunting in Elastic Stack: Packetbeat - pg. 60
Threat Hunting in Elastic Stack: Winlogbeat - pg. 63
Threat Hunting in Elastic Stack: ElasticAgent - pg. 65
NXLog - Amazing agent with both a Community and Enterprise (Premium) editions.
Free version has more features than many premium agents.
Has premium support available.
Supports tons of log formats including W3C logs, the log format of IIS, Exchange, and Bro/Zeek
For endpoints, can be installed on both Windows and Linux and have its log output in the same format.
Configuration Sections
Input - defines what logs to monitor for collection
Output - sets where logs should go and which protocol is used
Route - maps inputs and outputs
Extension Modules - converts between formats
NXLog-Autoconfig - With no customisation, the script will install Sysmon with the SwiftOnSecurity config, generate a NXLog config to start pulling the Sysmon and Windows Security events.
Syslog - Port 514. The most common protocol for logging. Can be TCP or UDP and can also be encrypted with TLS. Widely supported on most devices with the exception of Windows, which requires a third party agent.
Standard fields include: Time, Source, Facility, Severity, and Message.
Can also have a priority field associated with the log, calculated as Facilty(0-23) x 8 + severity (0-7)
Limitations: Size of logging packet is important especially when certain Windows logs can be over 30,000 bytes.
UDP - Limited to 1024 bytes, can lead to log duplication
TCP - Limited to 4096 bytes, is slower than UDP
Windows Event Forwarding - TCP Port 5985. Windows built in agent for forwarding Windows event logs, in standard EVTX format.
Can be managed by GPO: Group Policy Objects
Uses AD for authentication
Uses WinRM to push/pull logs to/from an event collector
Basic filtering, encryption, and compression
Agentless Log Collection - This usually involves a central server to collect logs, or a device that can natively send logs in the correct format to your SIEM.
Can use either a push or pull method with logs often being sent over WMI or SSH.
Easy to deploy and requires less maintenance
Can only handle a finite number of systems and logs
Can be a security risk with logs constantly being sent across the network and the potential requirement of the collector logging in to a device to pull the logs.
Data Diodes - A hardware or software tool that can improve log collection security in agentless applications, by forcing a one way connection to the logging server.
Script Collection
Sometimes the only method of collection for cloud and third party apps.
Handy for collecting baseline information and inventory
Log Aggregator
The central collection point for your logs. This is where raw logs can be parsed and event have context added to them.
Splunk Heavy Forwarder - A hybrid of a powerful logging agent and Aggregator, a Heavy forwarder can both collect logs from many sources as well as parse/filter logs.
Nagios - The Industry Standard In IT Infrastructure Monitoring - Network logging and monitoring tool focused for network engineering.
Logstash - One of the most common aggregators. Built by Elastic Stack with over 200 plugins. Can be used to accept logs and convert them to other formats used by premium SIEMs
Commercial support available
Uses a variety of plugins to perform different features
Can apply tags to logs and data for creating categories to enhance searching
HASecuritySolutions/logstash - Repo of tons of Logstash related material and configs
Plugins: Input - An input plugin enables a specific source of events to be read by Logstash.
Plugins: Codec - A Codec plugin will allow Logstash to read and parse certain special types of logging formats
Plugin: stdin - Allows for manual creation of logs
Plugin: File - Allows the output of files to be ingested as logs. Can be used as a way to log anything that can write a log to a file. Can use wildcards to scan for files in multiple directories.
Plugin: jdbc - This plugin was created as a way to ingest data in any database with a JDBC interface into Logstash. This is handy for certain endpoint security suites that store their logs in a database.
Plugin: UDP - The catch all plugin for logs being sent over syslog UDP 514. Can add TLS encryption to UDP streams. *Root privileges will be required on ports under 1024.*
Plugins: Output - Logstash can output to multiple different applications including Elasticsearch
Plugins: Filter - Filtering logs by tons of different options, variables, and conditions.
Regex - Filter by specific Regex matching
Grok - Filter by set Regex built patterns
Pattern - Filter by customer regex that is assigned a name. Reusable regex
CSV - Filter by comma separated values.
Key Value pair
JSON - Handy for parsing and filtering IDS solutions like SNORT and Surricata that can output in JSON format
Plugin: Date - Uses Joda-Time (java time library) to pattern match and normalize time/date stamps.
Plugin: Drop - Used for dropping logs buy certain criteria
Can also be used with the remove_field filter to remove specific data fields within a log
Plugins: Enrichment - Adding fields and contextual data.
GeoIP - Adds location to IP address entries
DNS - Performs reverse DNS lookups
Fingerprint - Takes a data field and replaces the data with a hash of the data itself. Handy for identification but not disclosure of sensitive data like PII.
Mutate - Manipulating a field and its data.
Ruby - For making API calls. Also for analysis of data fields like suspect length.
Log Broker
An optional component, a log broker acts as a buffer for temporary storage of logs. It helps to store logs over regular flow capacity. Intended to help syslog devices as many agent based systems have their own internal broker-like services.
Rabbitmq - Easy to install, easy to use, and available for all major operating systems
Kafka - High powered broker with the ability to manage high volume of events at a time. Use Apache Zookeeper to manage multiple Kafka nods
Storage
Where all of your logs are stored for later use.
WORM - Write-Once-Read-Many. The ideal storage type for SIEM
Typical relational database structures are not optimal as most are ACID compliant (Containing integrity checks) which is really not efficient for logging purposes.
Search/Report
The component which allows the searching of parsed data and creation of reports based on intricate search queries.
Elastisearch - The WORM based, distributed, scalable database used for real time searching.
Elastisearch architecture uses the concept of a shard, which is a storage partition. These shards are usually created with replicas stored in a different location for redundancy. The cool thing is that both the original shard, and the copy are available for searching, drastically increasing search speeds. Both the primary and replica shards are used to scale out.
Kibana - The Primary search interface for Elastisearch.
Threat Hunting in Elastic Stack: Using Kibana - pg. 66, 198
Threat Hunting in Elastic Stack: Bringing data into Elasticseach - pg. 50
Splunk - The premium cadillaic option of searching/data mining utilities.
Alert Engine
One of the most important components in a SIEM, the alert engine can take predefined searches and perform specific actions when those searches return results, such as create an alert.
Graylog - An admin platform for Elastisearch that can be used to create easy and flexible alerting
ElastAlert - A python based framework for setting up alerts within Elastisearch
Watcher/Kibana-Alerting - Elastisearch's commercial offering for creating and managing alerts.
Splunk Security Essentials - A free app(plugin) for Splunk that comes with a small set of security search use cases and other utilties.
Splunk Enterprise Security - Splunks premium SIEM alerting engine with a large selection of preset security alerts, deshboards, and an alert triaging panel for use in a SOC.
Last updated