TheHive, Cortex and MISP: How They All Fit Together

TheHive, Cortex and MISP work nicely together and if you’ve read our June-Dec 17 roadmap post, the integration of our products with the de facto threat sharing platform will get better in a few months.

During the FIRST conference presentation we gave last week, we displayed a picture that we will use here to try to explain how these three open source and free products integrate with one another.

Screen Shot 2017-06-16 at 09.58.43.png
A Picture is Worth a Thousand Words…

TheHive

TheHive is a Security Incident Response Platform (SIRP). It can receive alerts from different sources (SIEM, IDS, email. etc.) via its REST API. This is where alert feeders come into play.

Alert Feeders

Think of an alert feeder as a specialized program which consumes a security event (SIEM alert, email report, IDS alert, and so on), parses it and outputs an alert that its sends to TheHive through TheHive4py, the Python library we provide to interact with TheHive’s REST API.

We do not supply such feeders but developing them should be straightforward. If not, let us know  and we’ll do our best to help you out.

Alerts

Any alert sent to TheHive will show up in its Alerts pane. In addition to the sources mentioned above, new or updated MISP events will show up as well in that area if you configured TheHive to connect to one or several MISP instances. If so, TheHive will poll those MISP instance(s) at every interval looking for new or updated events. If there are any, TheHive will generate an alert which will end up in the Alerts pane.

Screen Shot 2017-06-18 at 15.29.51.png
The Alerts Pane

Alerts can be ignored, mark as read, previewed and imported. When an alert is imported, it becomes a case that needs to be investigated.

Cases

workflow
The Workflow that is at the Heart of TheHive

A case can be generated from an alert or created from scratch. It is subdivided into tasks (think identification, containment, eradication, check proxy logs, and so on) and observables (IP addresses, hashes, email addresses, domain names, URLs…). When analysts are working on tasks, they add logs as they go. In TheHive’s terminology, logs are text entries which may contain attachments to help analysts record what they have been doing. Logs can be written using Markdown or a rich-text editor.

Case Templates

You don’t need to add the same tasks over and over when working on cases belonging to a given category (DDoS, Malspam, APT, …). You can create custom templates to which you add tasks as shown below. This is very useful when you are dealing with alerts so that when you import them, you can select which case template you’d like to apply and there you go!

Screen Shot 2017-06-16 at 10.26.22.png
A Sample Case Template
Observables

Observables can be tagged, flagged as IOCs, and analyzed. When the investigation is well in progress or completed, you may want to share the resulting IOCs or a subset of those with partners and peers. TheHive will support the ability to export that data to MISP in September 2017. Until then, you can still export your IOCs as text, CSV or as a MISP-compatible format that you can use to add them to your MISP instance using the freetext editor. TheHive can export IOCs/observables in protected (hxxps://www[.]somewhere[.]com/) or unprotected mode.

Every observable must have a TLP (Traffic Light Protocol) level. By default, any added observable is considered TLP:AMBER. Please note that the TLP is taken into account by some analyzers. Wait! Analyzers?

Cortex

Cortex is our standalone analysis engine and a perfect companion for TheHive and MISP. Analysts can use it to analyze observables using its Web UI, in which case they can be submitted only one at a time. The Web UI should really be limited to quick assessments of observables before creating a case in TheHive (or in an alternate SIRP). The power of Cortex really comes into play when you use its REST API. TheHive speaks natively to Cortex (as MISP does). Moreover, TheHive can leverage one or several Cortex servers.

Screen Shot 2017-06-18 at 15.57.27.png
Observable Page and List of Analyzers
Analyzers

As of this writing, Cortex has 23 analyzers which come in a total of 39 flavors and more will be available soon.

An analyzer can be written in any programming language supported by Linux though all of our current analyzers are written in Python. This is because we provide a Python library called Cortexutils which contains a set of utility classes that make it easier to write an analyzer in Python.

Flavors

Analyzers such as VirusTotal, PassiveTotal or DomainTools can provide different analysis services. Let’s take VirusTotal as an example. You can scan a file or URL. That’s one flavor. You can also obtain the latest available report on VirusTotal.com for a file, hash, domain or IP address. That’s a second flavor. So the VirusTotal analyzer has two flavors.

Screen Shot 2017-06-18 at 16.26.41.png

How about PassiveTotal? It has 8 flavors: unique resolutions lookup, SSL certificate history lookup, malware lookup, passive DNS lookup, data enrichment lookup, SSL certificate details lookup, OSINT lookup and WHOIS data lookup.

The MISP Search Analyzer

At this point, we need to mention a special analyzer that may create some confusion if not understood correctly: the MISP Search analyzer. Thanks to it, Cortex has the ability to search observables within a MISP instance as represented by the arrow that goes from the Analyzers to MISP.

Screen_Shot_2017-06-19_at_08_03_54.png
Search for MISP Events Containing a Given Observable

When an observable is found in an event, Cortex will return the number of records found (i.e. the number of events where the observable has been found) and a list of links to those events with additional data.

Screen_Shot_2017-06-19_at_08_13_16.png
Searching for a Hash Using the MISP Search Analyzer from the Cortex Web UI
Screen Shot 2017-06-19 at 08.17.04.png
The Same Search Conducted from TheHive: Long Report
Screen Shot 2017-06-19 at 08.18.58.png
Mini-Report

The current version of the MISP Search analyzer can only search within a single MISP instance but in the near future, it will be able to support multiple ones.

MISP Expansion Modules

Besides its own analyzers (which include MISP Search described above), Cortex can also invoke MISP expansion modules. These are normally used by MISP to enrich attributes within events but Cortex can also take advantage of them to analyze observables.

There is some overlap between the native Cortex analyzers and MISP expansion modules. When choosing between a native analyzer or an expansion module, we highly recommend you select the former. The expansion modules are deactivated in the default Cortex configuration.

Jobs

When you submit an observable for analysis, Cortex will create a job and, if successful, it will generate an analysis report in JSON format. TheHive has the ability to parse those results and present them in a human-friendly fashion thanks to report templates we offer for free. So when you’ll submit an observable to Cortex from TheHive, you’ll get back a short (or mini) report and a long one. The first can be thought of as a really tiny Exec Analyst Summary while the second provides more insight and details.

Calling Cortex from MISP

In addition to the expansion modules we have just mentioned, MISP 2.4.73 and up can enrich attributes using Cortex analyzers. The configuration is pretty straightforward. So if all you are concerned about is threat intelligence and sharing, you may augment your visibility into a given threat represented as a MISP event by leveraging all current 23 Cortex analyzers and any future ones.

Conclusion

TheHive, Cortex and MISP are three open source and free products that can highly aid you combat threats and keep the ‘monsters’ at bay.

TheHive, as a SIRP, allows you to investigate security incident swiftly in a collaborative manner. Several analysts can work simultaneously on tasks & cases . While cases can be created from scratch, TheHive can receive alerts from different sources thanks to alert feeders which consume security events generated by multiple sources and feed them into TheHive using TheHive4py Python library. TheHive can also sync to one or several MISP instances to receive new and updated events which will appear in the alert pane with all the other alerts generated by other sources. Analysts can then preview new alerts to decide whether they need to be acted upon. If so, they can transform them into investigation cases using templates.

To analyze the observables collected in the course of an investigation and/or imported from a MISP event, TheHive can rely on one or several Cortex analysis engines. Cortex is another standalone product that we have developed which sole purpose is to allow you to analyze observables at scale thanks to its large number of analyzers, MISP expansion modules and any analyzer you might have developed on the side. Cortex has a REST API that can be used to empower other security products such as  ‘analytics’ software, alternate SIRPs or MISP.

The highly popular threat sharing platform can indeed enrich attributes thanks to Cortex as it has a native integration with it. And in a few months, you will also be able to export cases from TheHive as MISP events that you can share with peers and partners.

If you do share, you do care about our collective mission to defend the  digital assets that are under our watch from harm. So let us fight together as one.

 

 

Cortexutils 1.1.1 is Here

Lo and behold, The Chefs behind TheHive Project have been pretty busy at the code kitchen, continuously improving some of your favorite recipes. After releasing Mellifera 11.3 and Cortex 1.1.3, here comes the turn of Cortexutils, the Python library containing utility classes for Cortex analyzers.

Version 1.1.1 of the library solves an encoding issue, in fact a regression introduced by the previous release. It also corrects a situation where error reports wouldn’t be generated in case the analyzer did not require configuration (we have a handful of such creatures in store).

You can grab the new Cortexutils release through PIP. To update your existing installation, please run the following command:

 sudo pip install cortexutils --upgrade

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Cortex 1.1.3 Released

Moments ago, we have announced the release of Mellifera 11.3. And since we don’t want to leave you wanting for more update er fun time, you may want to schedule as well a Cortex update shall you need it 😉

Cortex 1.1.3 is now available. This hotfix corrects the deb package  to make it compatible with Ubuntu 16.04 without having to fiddle with OpenJDK. As we did with TheHive, we have repackaged the software to avoid grabbing OpenJDK 9 (which Cortex does not support) and force the installation of version 8. This version also corrects a cryptic error that might be thrown out by Cortex as a result of an improper interpretation of an analyzer failure.

Download & Get Down to Work

To update your current Cortex installation, follow the instructions of the installation guide. Before doing so, you may want to save the job reports that were not executed via TheHive. Cortex 1 has no persistence and restarting the service will wipe out any existing reports.

Please note that you can install Cortex using an RPM or DEB package, deploy it using an Ansible script, use Docker, install it from a binary or build it from sources.

Support

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

Cortex 1.1.2 Released

We are glad to announce a new version of your favorite observable analysis engine which corrects bugs introduced by version 1.1.1 and adds a few enhancements. As a reminder, TheHive, our Security Incident Response Platform, can interact with one or several Cortex instances. Moreover, starting from version 1.1.1, Cortex has a two-way integration with MISP.

We highly advise you to upgrade your Cortex in to instance to 1.1.2.

Screen Shot 2017-05-24 at 11.51.54.png
Cortex 1.1.2 – Job Report Example with CERT-SG’s Abuse Finder

Fixed Issues

  • #27: fixed the daunting error 500 that many users of  TheHive encountered when a job is submitted to Cortex.
  • #29: the MISP expansion modules are now disabled by default to avoid another error 500.
  • #31: the web interface was displaying SNAPSHOT (oops!) for the Cortex version.  It now displays the correct version.

Enhancements

  • #28: when you enable the MISP expansion modules, Cortex will not be slowed down and starts without delay.
  • #30: add a page loader mask similar to TheHive’s.

Download & Get Down to Work

To update your current Cortex installation, follow the instructions of the installation guide. Before doing so, you may want to save the job reports that were not executed via TheHive. Cortex 1 has no persistence and restarting the service will wipe out any existing reports.

Please note that you can install Cortex using an RPM or DEB package, deploy it using an Ansible script, use Docker, install it from a binary or build it from sources.

Support

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

VMRay, FireHOL, Joe Sandbox & Fortiguard Analyzers

We are pleased to announce the availability of 2 new Cortex analyzers and an update to 2 existing ones:

  • New: VMRay and FireHOL
  • Updated: Joe Sandbox and Fortiguard URL Category

We would like to thank Nils Kuhnert from CERT-BUND, CERT-BDF and Eric Capuano for their precious contributions.

To install the new analyzers, grab the Cortex-Analyzers repository and unpack its content (or git pull the master  branch) in your existing /path/to/cortex-analyzers. Then follow the Cortex analyzers guide.

To import the new report templates in your instance of TheHive:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu
  • click on Import templates button and select the downloaded package

VMRay

The VMRay analyzer has been submitted by Nils Kunhert from CERT-BUND. It lets you run a file in a local or remote (cloud) VMRay sandbox. The analyzer also lets you check existing analysis reports.

The analyzer accepts files and hashes as input. VMRay is a commercial service and you need an API key to run the analyzer. To make it work, install the requests Python library. It should already have been installed since it is used by other analyzers as well.

To use the analyzer, add the following section to the Cortex configuration file (application.conf):

VMRay {
 url = ""
 key = ""
 certpath = ""
 }

When called from TheHive, the following output is produced:

sc-short-vmray.png

sc-long-vmray.png
TheHive: VMRay Analyzer – Short and Long Report Samples

Important note: an analysis on VMRay, like on any other sandbox, can take a long time. That is why the analyzer tries to fetch the report until it is ready.

FireHOL

The FireHOL analyzer has been submitted by Nils Kuhnert from CERT-BUND. It lets you use the lists maintained by FireHOL project and check if an IP resides in one of them. FireHOL is an open source project. The analyzer reports the block lists in which an IP resides with the latest updated ones displayed first. To make it work, you’ll need to download the lists in a directory first (and it would be wise to do it on a regular fashion using a cron entry for example):

git clone https://github.com/firehol/blocklist-ipsets

The FireHOL analyzer depends on the following librairies:

ipaddress
pytz
dateutil
datetime

Add the following section to the Cortex configuration file (application.conf) to activate the analyzer:

FireHOLBlocklists {
 blocklistpath = ""
 ignoreolderthandays = <int>
 }

The ignoreolderthandays parameter lets you tell the analyzer to ignore matches found in lists that have not been refreshed in <int> days where <int> is an integer.

When called from TheHive, the following output is produced:

sc-short-fireHOL.png

sc-long-firehol.png

TheHive: FireHOL Analyzer – Short and Long Report Samples

Joe Sandbox

Thanks to CERT-BDF, the Joe Sandbox analyzer has been updated to support Joe Sandbox Cloud service beside the on-premises version (Ultimate). Like with other Joe Sandbox services, you need to add the following section to the Cortex configuration file (application.conf):

JoeSandbox {
 url = ""
 apikey = ""
 }

Fortiguard URL Category

Thanks to Eric Capuano, the Fortiguard URL Category analyzer is working again. Eric has modified it to handle the changes made by Fortiguard to their free online API.

Correction: May 23, 2017
An earlier version of this post used ignoredays instead of ignoreolderthandays for the FireHOL Blocklists analyzer. This parameter has also been described.

Cortex 1.1.1: Two Way MISP Integration Now a Reality

TheHive Project’s Chefs are thrilled to announce the immediate availability of Cortex 1.1.1.

Starting from this version, Cortex can be integrated in two ways with MISP as described below. We would like to thank Alexandre Dulaunoy for inviting us to the Open Source Security Software Hackathon which took place in Luxembourg during two days (May 2-3, 2017). Andras Iklody worked with us during the event in order to make this two way integration a reality. Merci !

Now in addition to TheHive, our Security Incident Response Platform which can connect to multiple MISP instances to receive new or updated events, let analysts preview then import them if they deem them worth investigating, Cortex can query MISP modules or be invoked from MISP to let an instance’s users leverage the power of its 21 analyzers. And in the near future, TheHive will also gain the ability to export observables to MISP.

Invoke MISP modules within Cortex

Besides its regular analyzers, Cortex 1.1.1  can analyze observables using MISP expansion modules.

Obviously, there are some overlap between Cortex native analyzers and MISP expansion modules. For example, you could query the CIRCL’s Passive DNS service using a native Cortex analyzer or a MISP expansion module. When there’s overlap, we highly recommend you rely on the Cortex analyzer. That way, we will be able to better help you in case you encounter issues or need help to make it work.

In order to invoke MISP expansion modules within Cortex, they need to be installed on the same host that Cortex runs on. Please read the MISP Integration guide.

Invoke Cortex Analyzers within MISP

Starting from version 2.4.73, a MISP instance can invoke Cortex analyzers. To do so, connect to the MISP Web UI with sufficient privileges, then go to Administration > Server settings > Plugin settings. Edit the Cortex section as follows:

  • set Plugin.Cortex_services_enable to true
  • set Plugin.Cortex_services_url  to http://ip_address (replace ip_address with the IP address of Cortex)
  • set Plugin.Plugin.Cortex_services_port to port (replace port with the port on which Cortex is listening: 9000 by default)

Once this operation is completed, the Cortex analyzer list should appear in MISP’s Cortex section. The analyzers must be enabled to make them available to the instance users.

Documentation

Please note that we have moved all the documentation of Cortex to a new repository.

Download & Get Down to Work

To update your current Cortex installation, follow the instructions of the installation guide. Before doing so, you may want to save the job reports that were not executed via TheHive. Cortex 1 has no persistence and restarting the service will wipe out any existing reports.

Please note that you can install Cortex using an RPM or DEB package, deploy it using an Ansible script, use Docker, install it from a binary or build it from sources.

Support

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

Correction: May 18, 2017
An earlier version of this post contained a few typos which were corrected.

Cortex Passes the 20 Analyzers Mark

Thanks to the invaluable contributions of our growing and thriving user community, Cortex has now 6 more analyzers, bringing the total to 21. The new analyzers, released under our usual AGPL v3 license, are:

  • CIRCLPassiveDNS
  • CIRCLPassiveSSL
  • GoogleSafebrowsing
  • Nessus
  • Virusshare
  • Yara

All but one have been submitted by Nils Kuhnert of CERT-Bund. The Nessus analyzer has been contributed by our long-time friend Guillaume Rousse.

Cortexutils 1.1.0

While reviewing the submissions, we realized that a new version of the Cortexutils library was needed in order to support both Python 2 and 3. Hence we released version 1.1.0. You can grab it through PIP. To update your existing installation, please run the following command:

 sudo pip install cortexutils --upgrade

Note that Cortexutils 1.1.0 is required to benefit from these analyzers and future ones. If you are performing a fresh Cortex installation, follow the guide.

Installation

To install the new analyzers, grab the Cortex-Analyzers repository and unpack its content (or git pull the master  branch) in your existing /path/to/cortex-analyzers. Then follow the Cortex analyzers guide.

New Short and Long Report Templates for TheHive

Short and long reports for TheHive were also created to parse and display the results produced by the new analyzers. We also bundled in the new package a URL analysis template for Joe Sandbox which was missing and improved some of the older short templates in order to follow a taxonomy.

To import the new report templates in your instance of TheHive:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu
  • click on Import templates button and select the downloaded package

CIRCLPassiveDNS

The CIRCLPassiveDNS analyzer lets you check the CIRCL’s Passive DNS service for a given domain. It takes domains and URLs as input. Access to the service is allowed to trusted partners in Luxembourg and abroad. If you think you qualify, please contact the good CIRCL folks. To make it work, you’ll need the pypdns Python library.

In order to take advantage of CIRCLPassiveDNS, you need to add the following section to the Cortex configuration file (application.conf):

CIRCLPassiveDNS {
     user=""
     password=""
}

When called from TheHive, the following output is produced:

sc-short-CIRCLPassiveDNS.png

sc-long-CIRCLPassiveDNS.png
TheHive: CIRCLPassiveDNS Analyzer – Short and Long Report Samples

CIRCLPassiveSSL

The CIRCLPassiveSSL analyzer lets you check CIRCL’s Passive SSL service for a given IP address or certificate hash. Access to the service is restricted to partners and security researchers worldwide. If you think you qualify, please contact the good CIRCL folks. This analyzer needs the pypssl Python library to work properly.

To use it, please add the following section to the Cortex configuration file (application.conf):

CIRCLPassiveSSL {
     user=""
     password=""
}

When called from TheHive, the following output is produced:

sc-short-CIRCLPassiveSSL.png

sc-long-CIRCLPassiveSSL.png
TheHive: CIRCLPassiveSSL Analyzer – Short and Long Report Samples

GoogleSafebrowsing

This analyzer lets you check URLs and domains against Google Safe Browsing. You need an API key to use it.

To leverage GoogleSafebrowsing, add the following section to Cortex’ configuration file:

GoogleSafebrowsing {
  key="" 
}

When you run the analyzer fromTheHive, you should see output similar to the samples below:

sc-short-safebrowsing.png

sc-long-safebrowsing.png
TheHive: GoogleSafebrowsing Analyzer — Short and Long Report Samples

Nessus

The Nessus analyzer lets you leverage Tenable’s Nessus Vulnerability Scanner to scan an IP address or a FQDN. Of course, you must not scan assets that do not belong to you, unless you really know what you are doing. That’s why safeguards were built in the analyzer’s configuration:

Nessus {
   url="<https://url.to.scanner>"
   login=""
   password=""
   policy=""
   ca_bundle=""
   allowed_networks=[ 'x.y.z.t/8', 'a.b.c.d/24', ... ]

The nessrest Python library is needed to make REST API calls to Nessus. Analysts would use the analyzer to assess the vulnerabilities of potentially compromised machines or new, unknown assets that have been plugged into one of their constituency’s networks. Of course, penetration testers conducting large-scale reconnaissance can also benefit from this analyzer.

sc-short-nessus.png

sc-long-nessus.png
TheHive: Nessus Analyzer — Short and Long Report Samples

Virusshare

The Virusshare analyzer lets you verify whether a file or hash is available on VirusShare.com. It requires the progressbar2 Python library besides requests (which should be already installed if you have an existing Cortex installation). As stated by Nils:

This analyzer enables searching for md5 hashes in Virusshare.com hash list. It does not download samples for you nor links directly to the sample – the author of virusshare prohibits the automatic download/site scraping and I respect that. It provides a button to start the virusshare search, though, but you need an account for that. You can request an invitation to the platform through contacting the admin via mail, directly.

To use it, add the following section to your Cortex application.conf:

Virusshare {
   path="/path/to/download/directory"
}

Quoting Nils again, in order to download the newest available hash lists from virusshare.com, you can run the download_hashes.py script that comes with the analyzer.

./download_hashes.py /path/to/your/download/directory

Upon running the analyzer from TheHive, the report will contain a link to the corresponding Virusshare page if a match is found as shown below.

sc-long-virusshare.png
TheHive: Virusshare Analyzer — Long Report Sample

Yara

Last but not least, the Yara analyzer can check files against YARA rules using yara-python. To use it, add the following to your Cortex configuration file:

Yara {
    rules=["/path/a", "/path/b", "/path/my/rules.yar"]
}

You can specify path to directories and files. If you supply a directory, the analyzer expects to find an index.yar or index.yas file. The index file can include other rule files. An example can be found in the Yara-rules repository.

sc-short-yara.png

sc-long-yara.png
TheHive: Yara Analyzer — Short and Long Report Samples

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!