HELK – An Open Source Threat Hunting Platform

The Hunting ELK or simply the HELK is an Open Source Threat Hunting Platform with advanced analytics capabilities such as SQL declarative language, graphing, structured streaming, and even machine learning via Jupyter notebooks and Apache Spark over an ELK stack.

This project was developed primarily for research, but due to its flexible design and core components, it can be deployed in larger environments with the right configurations and scalable infrastructure.

Goals of HELK Open Source Threat Hunting Platform

• Provide an open-source hunting platform to the community and share the basics of Threat Hunting.
• Expedite the time it takes to deploy a hunting platform.
• Improve the testing and development of hunting use cases in an easier and more affordable way.
• Enable Data Science capabilities while analyzing data via Apache Spark, GraphFrames & Jupyter Notebooks.

Installing HELK

Requirements (Please Read Carefully)

• Operating System:
  • Ubuntu 18.04 (preferred)
  • Ubuntu 16
  • CentOS 7 with or without SELinux in enforcement mode
  • CentOS 8 with or without SELinux in enforcement mode
• Docker:
  • HELK uses the official Docker Community Edition (CE) bash script (Edge Version) to install Docker for you. The Docker CE Edge script supports the following distros: ubuntu, debian, raspbian, centos, and fedora.
  • You can see the specific distro versions supported in the script here.
  • If you have Docker & Docker-Compose already installed in your system, make sure you uninstall them to avoid old incompatible version. Let HELK use the official Docker CE Edge script execution to install Docker.
• Processor/OS Architecture:
  • 64-bit also known as x64, x86_64, AMD64 or Intel 64.
  • FYI: old processors don’t support SSE3 instructions to start ML (Machine Learning) on elasticsearch. Since version 6.1 Elastic has been compiling the ML programs on the assumption that SSE4.2 instructions are available (See: https://github.com/Cyb3rWard0g/HELK/issues/321 and https://discuss.elastic.co/t/failed-to-start-machine-learning-on-elasticsearch-7-0-0/178216/7)
• Cores: Minimum of 4 cores (whether logical or physical)
• Network Connection: NAT or Bridge
  • IP version 4 address. IPv6 has not been tested yet.
  • If using a proxy, documentation is yet to come – so use a proxy at your own expense. However, open a GitHub issue and we will try to help until it is officially documented/supported.
  • If using a VM then NAT or Bridge will work.
  • Internet access
    • List of required domains/IPs will be listed in future documentation.
• RAM: There are four options, and the following are minimum requirements (include more if you are able).
  • Option 1: 5GB includes KAFKA + KSQL + ELK + NGNIX.
  • Option 2: 5GB includes KAFKA + KSQL + ELK + NGNIX + ELASTALERT
  • Option 3: 7GB includes KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER.
  • Option 4: 8GB includes KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER + ELASTALERT.
• Disk: 20GB for testing purposes and 100GB+ for production (minimum)
• Applications:
  • Docker: 18.06.1-ce+ & Docker-Compose (HELK INSTALLS THIS FOR YOU)
  • Winlogbeat running on your endpoints or centralized WEF server (that your endpoints are forwarding to).
    • You can install Winlogbeat by following one of @Cyb3rWard0g posts here.
  • Winlogbeat config recommended by the HELK since it uses the Kafka output plugin and it is already pointing to the right ports with recommended options. You will just have to add your HELK’s IP address.

HELK Download

Run the following commands to clone the HELK repo via git.

git clone https://github.com/Cyb3rWard0g/HELK.git

HELK Install

In order to make the installation of the HELK easy for everyone, the project comes with an install script named helk_install.sh. This script builds and runs everything for HELK automatically. During the installation process, the script will allow you to set up the following:

• Set the components/applications for the HELK’
• Set the Kibana User’s password. Default user is helk
• Set the HELK’s IP. By default you can confirm that you want to use your HOST IP address for the HELK, unless you want to use a different one. Press [Return] or let the script continue on its own (90 Seconds sleep).
• Set the HELK’s License Subscription. By default the HELK has the basic subscription selected. You can set it to trial if you want and will be valid for 30 days. If you want to learn more about subscriptions go here
  • If the license is set to trial, HELK asks you to set the password for the elastic account.

To install HELK:
Change your current directory location to the new HELK directory, and run the helk_install.sh bash script as shown:

cd HELK/docker
sudo ./helk_install.sh

Read more here.