This project is designed for users seeking to monitor and analyze network infrastructure activity, enhance privacy controls, or address complex security challenges. It supports scalable deployment, enabling configurations as an Intrusion Detection System (IDS) or an Intrusion Prevention System (IPS). The integrated Security Information and Event Management (SIEM) system provides centralized log aggregation, real-time event correlation, and in-depth historical analysis. The IDS leverages rule-based and signature-based detection mechanisms to identify potential threats or anomalous events, facilitating proactive threat management.
SIEM-IDS integrates open source tools to monitor, analyze, correlate, and alert on network traffic, systems, applications, and security events in real-time. It offers detailed visualizations of every activity, providing actionable insights into your privacy and security posture.
Disclaimer: This is currently in development and is safe to run in a staging environment or as a demo. However, it is not recommended for production use due to incomplete features, such as the under-development Suricata log shipping pipeline on Vector and partial integration with the OTX API | Threat Intelligence feature.
- Suricata is a powerful Network Intrusion Detection System (IDS), Intrusion Prevention System (IPS), and Network Security Monitoring (NSM) engine, it captures and inspects network traffic.
- The captured data, formatted in the Extensible Event Format (EVE), is processed, enriched, and sent to Elasticsearch by Filebeat. In addition to Suricata logs, Filebeat collects and forwards logs from auditd, auth.log, syslog, and the local DNS server (e.g., Pi-hole or, alternatively Blocky).
- Vector handles the collection and forwarding of Suricata's EVE logs to a dedicated index
suricata-*, keeping it separate from the generallogs-*index for better organization and analysis."
The Vector pipeline is to integrate the OTX (Open Threat Exchange) feature and is currently under development.
- Docker/Podman Compose Configuration
- Docker/Podman Container Security
- ELK Configuration and Objects
- ELK Passwords and Secrets
- Filebeat Log Collection and Enrichment
- Before running ./start.sh
- Ready to start
- Traffic Flow Map
- OTX Open Threat Exchange (under development)
- Screenshots
- Contributing
- License
- Clone the Repository:
git clone https://github.com/virtueistheonlygood/siemids
cd siemids- Install Podman/Docker:
This project requires either Podman/Compose or Docker/Compose to run.
- Podman: Follow the installation instructions on the Podman website and the podman-compose GitHub repository..
- Docker: Install Docker by following the guidelines on the Docker website and check out the docker-compose GitHub repository for more details.
Suricata requires a tailored configuration to work correctly on your infrastructure, whether running as an IDS or IPS. Ensure that the configuration paths align with your installation. In this project, we use a containerized version of Suricata 7.0.8.
- We strongly recommend compiling and running Suricata on your host by customizing the configuration based on your network topology and monitoring needs. For reference, use the sample configuration file
resources/suricata/suricata.yaml.
You can follow the instructions in the Suricata documentation to compile and install it on your system.
Key configuration aspects include:
- Network Interfaces: Specify the network interfaces that Suricata should monitor for traffic analysis
af-packet:
- interface: wlp3s0
threads: auto
cluster-id: 99
cluster-type: cluster_flow
defrag: yes
use-mmap: yes
mmap-locked: yes
tpacket-v3: yes
ring-size: 2048
block-size: 32768
block-timeout: 10
use-emergency-flush: yes
buffer-size: 32768
disable-promisc: no
checksum-checks: kernel
- interface: wg0
threads: auto
cluster-id: 98
cluster-type: cluster_flow
defrag: yes
use-mmap: yes
mmap-locked: yes
tpacket-v3: yes
ring-size: 2048
block-size: 32768
block-timeout: 10
use-emergency-flush: yes
buffer-size: 32768
disable-promisc: no
checksum-checks: kernel- Rule Sets: Specify the rule sets that Suricata should use to detect threats.
default-rule-path: /usr/local/var/lib/suricata/rules
rule-files:
- suricata.rules- Logging: Configure logging options to capture and store alerts and other relevant data. This is the path used by Filebeat and Vector to collect and ship Suricata EVE logs to Elasticsearch.
default-log-dir: /suricata/The docker-compose.yaml or podman-compose.yaml files orchestrate the deployment of the various services needed. It specifies the configuration for each service, including security, network settings, environment variables, and volume mounts, ensuring a seamless integration of all components.
Communication between the containerized services (Elasticsearch, Filebeat, and Kibana) is secured using HTTPS/SSL, with a Public Key Infrastructure (PKI) implemented by the 'setup_pki' service. PKI employs a pair of cryptographic keys (public and private) to authenticate the entities involved and encrypt data in transit, ensuring that sensitive information is securely transmitted and accessible only to authorized parties.
setup_pki:
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- ./certs:/usr/share/elasticsearch/config/certs
user: "0"- The certificates used for securing communication between containerized services are stored under the
certs/directory.
The visualizations and dashboards are stored in saved.objects/siemids.ndjson and imported to kibana by the setup_kibana as soon as the kibana service is healthy.
command: >
sh -c '
until curl -s -I --cacert /usr/local/certs/ca/ca.crt https://localhost:5601 | grep -q "HTTP/1.1 302 Found"; do
echo "Waiting for Kibana to be ready...";
sleep 10;
done;
curl -s -X POST "https://localhost:5601/api/saved_objects/_import?compatibilityMode=true" -H "kbn-xsrf: true" --cacert /usr/local/certs/ca/ca.crt --cert /usr/local/certs/elasticsearch/elasticsearch.crt --key /usr/local/certs/elasticsearch/elasticsearch.key -u elastic:${ELASTIC_PASSWORD} --form file=@/usr/local/saved.objects/siemids.ndjson
'- Configuration files are located in
resources/directory
Passwords are randomly generated and stored as secrets and in the .env file. Use the ELASTIC_PASSWORD to log in at https://localhost:5601. After securing your environment and passwords, be sure to delete the .env file.
secrets:
- elastic_password
- kibana_password
command: >
bash -c '
ELASTIC_PASSWORD=$(cat /run/secrets/elastic_password)
KIBANA_PASSWORD=$(cat /run/secrets/kibana_password)
if [ x${ELASTIC_PASSWORD} == x ]; then
echo "Set the ELASTIC_PASSWORD secret";
exit 1;
elif [ x${KIBANA_PASSWORD} == x ]; then
echo "Set the KIBANA_PASSWORD secret";
exit 1;
fi;Ensure that the Filebeat configuration, including file permissions, file paths, processors, fields, and dissect definitions, is properly set up to meet your environment's requirements. Refer to the official Filebeat documentation for additional details.
nano resources/filebeat/filebeat-siemids.ymlSystem logs are collected from the standard path /var/log, including files such as syslog, auth.log, and audit.log.
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/pihole/pihole.log
- /var/log/pihole/pihole-FTL.log
fields:
event.dataset: pihole.log
fields_under_root: true
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: true
# Period on which files under path should be checked for changes
reload.period: 120s
filebeat.modules:
- module: auditd
log:
enabled: true
var.paths: ["/var/log/audit/audit.log*"]
- module: system
syslog:
enabled: true
var.paths: ["/var/log/syslog*"]
auth:
enabled: true
var.paths: ["/var/log/auth.log*"]As mentioned earlier, Suricata EVE logs are written to /suricata and collected by Filebeat and Vector.
- module: suricata
eve:
enabled: true
var.paths: ["/suricata/eve.json"][sources.suricata]
type = "file"
include = ["/suricata/eve.json"]
read_from = "end"
fingerprint.strategy = "device_and_inode"
ignore_checkpoints = false # Ensure checkpoints are used to avoid re-ingesting logsThe final requirement is to manually modify resources/interface.py to define the names of your local and/or VPN interfaces. This step is necessary for enriching Suricata EVE logs with GeoIP information for both your local and public networks in Kibana visualizations. The enrichment, which supports complex traffic flow maps, is handled by Filebeat. It loads the data from the geoip/*.json and env/*.env files, which are generated by this python script.
localnet_1 = "wlp3s0" # modify with your local interface name
vpnet_1 = "wg0" # modify with your vpn interface name
#localnet_2 = "wlp3s1" # modify with your local interface name
#vpnet_2 = "wg1" # modify with your vpn interface name- We are planning to automate this step by identifying active network interfaces.
Note: Running the start.sh script will not affect your running containers, however it will clean your local environment from previous setups and set system parameters required for Elasticsearch on your host and change ownership and permissions of configuration files in resources/. This ensures that any previous configurations or data do not interfere with the new setup.
sudo ./start.shYes, you need sudo to ensure that Podman has the necessary permissions.
- Podman need elevated privileges to perform certain tasks, such as accessing network interfaces, binding to ports below 1024 and due the configuration of certain capabilities (e.g., NET_ADMIN), mounting directories and files that require root access.
After completing the previous steps, we can now visualize network events using the custom visualizations created for this project, as shown in this screenshot of the Event Visualization Map, highlighting traffic flow and source/destination details:
Add your OTX AlienVault API_KEY to resources/otx-api.py
API_KEY = 'YOUR_OTX_API_KEYTo query the OTX API you can use tools like curl and jq.
curl "http://127.0.0.1:5000/threat_intel?ip=<IP_ADDRESS>" | jq .Replace <IP_ADDRESS> with the IP address you want to query. For example:
curl "http://127.0.0.1:5000/threat_intel?ip=8.8.8.8" | jq .
Contributions are welcome! Please feel free to fork the repository and submit a pull request with your changes.
This project incorporates components licensed under various open-source licenses:
- Suricata: Licensed under the GPLv2 License.
- Elasticsearch: Licensed under the Elastic License 2.0.
- Kibana: Licensed under the Elastic License 2.0.
- Filebeat: Licensed under the Elastic License 2.0.
- Vector: Licensed under the Apache License 2.0.
- Pi-Hole: Licensed under the GPLv3 License.
Please refer to the respective project’s LICENSE file for more detailed information. Feature