Cerana 0.9 and Cortex 2.0.4 are Out!

We are proud to announce the immediate availability of Cerana 0.9 (TheHive 3.0.9) and Cortex 2.0.4. These hotfix releases address a number of issues and we encourage you to update your current installation at your earliest opportunity. For your comfort and sanity. Seriously.

We also took this opportunity to update Cortex analyzers to fix issues with CIRCL Passive SSL, Hybrid Analysis, and the Joe Sandbox URL Analysis template. Moreover, we have updated the cortexutils library to set the taxonomy level to info if it is invalid. To upgrade cortexutils​ to 1.2.4:

sudo pip install -U cortexutils && sudo pip3 install -U cortexutils

To update your Cortex analyzers:

cd /path/to/Cortex-analyzers && git pull

Note: the Bluecoat analyzer was removed since it does not comply with the updated Terms of Service of Symantec Web Pulse SiteReview. Symantec does no longer permit programmatic querying of the service.

Fixes in Cerana 0.9

  • #527: display long reports when the analyst clicks on the corresponding short reports. Meh!
  • #541: make the drop-down menu for case templates scroll when there is a truckload of them.
  • #452: prevent WSAPI failure.
  • #531: fix naming inconsistencies in the Live Stream.
  • #530: correct an error when trying to analyze a filename using the Hybrid Analysis analyzer.
  • #543: generate an error if unable to contact Cortex.
  • #518: merge observable sightings when merging cases.
  • #535: fix the tag color of the PhishTank analyzer which was transparent under certain conditions.

Fixes in Cortex 2.0.4

  • #89: let a read,analyze user change or display their API key.
  • #91: sort analyzers by name.
  • #92: redirect users to the index page when they click on the Cortex logo.
  • #93: under the Organization > Configurations page, the UI displays wrong green checkmarks for empty configurations.
  • #94: orgadmin users are not able to update their organization’s users after the users are created. The UI doesn’t display any error message.
  • #95: avoid ‘lax programming’, Nabil style😜, and strictly filter the list of analyzers in the Run dialog.
  • #90: fix Python dependency errors in docker.

Support

Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

CorrectionApril 14, 2018
An earlier version of this post did not mention that the Bluecoat analyzer was removed in the latest Cortex Analyzers repository release.

Cortex 2, TheHive and a Whole Slew of Updates

After announcing Cortex 2.0.0 and TheHive 3.0.7, the first version of your favorite SIRP that is (supposedly) compatible with the brand-new version of Cortex, last week, we thought it was time to relax and enjoy the upcoming, long Easter weekend, the sunny sky of Paris (if you can pierce the veil of the Forever Grey Cloud™ that is hanging over the city of lights), and great jazz music. Heck, I even tweeted about it … only to be proven wrong by Life (and Murphy).

We literally field tested Cortex 2 for 3 weeks, we squashed bugs here and there, until almost the very last minute before the release. And yet, our QA needs to be improved by leaps and bounds as we had to release Cortex 2.0.1 one day after unveiling 2.0.0 to correct some additional bugs. And then some members of the core team and of our growing user community took it for a spin. And all hell broke lose. Well, almost 🙂

good_code
Source: XKCD

Session collisions (when TheHive and Cortex 2 are used on the same machine), analyzer malfunctions, connectivity problems … issues that were not identified during the testing phase, even in a production environment, where everything worked as expected. And we call this ‘Computer Science’. Right, right…

So we worked hard, took out our Code Hammer (it’s like Thor’s but cyber) and blasted away all the bugs that we found out or that were reported to us (arigato gozaimasu!) and we are happy to announce the immediate availability of Cortex 2.0.2, TheHive 3.0.8, Cortexutils 1.2.3 and Cortex-Analyzers 1.9.2.

TL;DR Install or upgrade Cortex 2.0.2, update Cortexutils, git pull the Cortex-analyzers repo to get the latest version of the repository, upgrade to TheHive 3.0.8, follow the Quick Start Guide and have a drink.

If you have time (which is admittedly quite scarce nowadays), please read on the changelogs:

What’s Next?

As stated in the previous post, we will release a new version of Cortex4py in order to make it compatible with Cortex 2, continue the work we started with our MISP Project friends to support MISP attribute enrichment through Cortex 2 (MISP currently only supports enrichment using Cortex 1), and perform a long-overdue overhaul of our documentation. We will also release a brand new version of TheHive4py.

Last but not least, we’ll take a hard look at ourselves and our QA. You expect us from us high quality and we hold ourselves to high standards. And we will deliver.

Support

Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

All Fresh CortexUtils, New Cortex Analyzers

Ahead of the imminent release of Mellifera 12 (TheHive 2.12.0), a new, major (as in MAJOR) version of your (soon to be?) favorite Security Incident Response Platform, we’ve made rather significant changes to Cortex analyzers and released a new version of the CortexUtils Python library.

TL;DR

If you are in a hurry:

$ sudo pip install cortexutils --upgrade
$ cd where/your/analyzers/are
$ git pull master

Adjust the Cortex configuration for the new MISP 2.0 analyzer and for Hippocampe as shown below if you are using these analyzers then import the corresponding report templates into TheHive:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu
  • click on Import templates button and select the downloaded package

CortexUtils 1.2.0

CortexUtils has been updated to include a new function called build_taxonomy() which is required for analyzers relying on the Python library we released to make their development development easier.

Mini-Reports in The Observable Tabs

Starting from Mellifera 12 (TheHive 2.12.0), mini-reports will be displayed in the observable tab in each case as soon as an analysis has been completed. Now analyzers compute their short/mini reports and put them in the summary section of their JSON output, ready for consumption. TheHive 2.12.0 and up will no longer create them on-the-fly.

Taxonomy

The mini-reports of all the analyzers have been updated to comply with a taxonomy that is similar to the one we were already using for VirusTotal:  VT:Score="14/56”.  A “maliciousness” level was already included in TheHive’s analyzer templates and we used a specific color to display each level. This level is now produced directly by the analyzers:

  • info / blue: the analyzer produced an information, and the short report is shown in blue color in TheHive.
  • safe / green : the analyzer did not find anything suspicious or the analyzed observable is safe (according to the analyzer). TheHive displays the short report in green color.
  • suspicious / orange : the analyzer found that the observable is either suspicious or warrants further investigation. The short report is orange colored in TheHive.
  • malicious / red : the analyzer found that the observable is malicious. The short report is displayed by TheHive in red color as show below:

sc-short-VT.png

The short report is built with the summary() function of an analyzer. The build_taxonomy() of cortexutils mentioned earlier should help building it.

MISP 2.0

The MISP analyzer has been updated to version 2.0 and includes new functionality submitted by our long-term contributor Nils Kuhnert from CERT-Bund (thanks a heap!). Unlike the previous version, v 2.0 will let you search for an observable in multiple MISP servers at the same time.

The analyzer accepts a truckload of datatypes as input. To make it work, install the pymisp Python library. It should already have been installed if you are just updating your current analyzers. You will also have to change Cortex configuration file (application.conf) for this new version:

MISP {
 url=["https://mymispserver_1", "https://mymispserver_2"]
 key=["mykey_1", "mykey_2" ]
 certpath=["", ""]
 name=["MISP_SERVER_NAME_1", "MISP_SERVER_NAME_2"]
}

Important note: You have to adjust your existing configuration to match the one shown above. The certpath variable can be left blank if you are not using a self-signed certificate.

When called from TheHive, the following output is produced:

sc-short-MISP.png

sc-MISP-long.png
TheHive: MISP 2.0 Analyzer – Short and Long Report Samples

The short report will show the number of unique events found in all MISP servers while the long report will show information of each matching event in each MISP server.

CERTatPassiveDNS

The CERTatPassiveDNS analyzer is a new submission by Nils (thanks again). It lets you check the CERT.at PassiveDNS service for a given domain or hostname. It takes domains and FQDN as input.

Access to the CERT.at service is allowed to trusted partners only. If you think you qualify, please contact CERT.at. You do not need to add specific information into the Cortex configuration file to benefit from this analyzer as it calls the whois system command to perform the pDNS requests.

When called from TheHive, the following output is produced:

sc-certatpdns-short.png

 

sc-long-CERTatPassiveDNS_1_0.png
TheHive: CERTatPassiveDNS Analyzer – Short and Long Report Samples

 

Miscellaneous

The latest version of the Cortex-analyzers repository also include the following bug fixes and improvements:

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Correction: July 6, 2017 
An earlier version of this post mispelled Nils Kuhnert’s last name.

TheHive, Cortex and MISP: How They All Fit Together

TheHive, Cortex and MISP work nicely together and if you’ve read our June-Dec 17 roadmap post, the integration of our products with the de facto threat sharing platform will get better in a few months.

During the FIRST conference presentation we gave last week, we displayed a picture that we will use here to try to explain how these three open source and free products integrate with one another.

Screen Shot 2017-06-16 at 09.58.43.png
A Picture is Worth a Thousand Words…

TheHive

TheHive is a Security Incident Response Platform (SIRP). It can receive alerts from different sources (SIEM, IDS, email. etc.) via its REST API. This is where alert feeders come into play.

Alert Feeders

Think of an alert feeder as a specialized program which consumes a security event (SIEM alert, email report, IDS alert, and so on), parses it and outputs an alert that its sends to TheHive through TheHive4py, the Python library we provide to interact with TheHive’s REST API.

We do not supply such feeders but developing them should be straightforward. If not, let us know  and we’ll do our best to help you out.

Alerts

Any alert sent to TheHive will show up in its Alerts pane. In addition to the sources mentioned above, new or updated MISP events will show up as well in that area if you configured TheHive to connect to one or several MISP instances. If so, TheHive will poll those MISP instance(s) at every interval looking for new or updated events. If there are any, TheHive will generate an alert which will end up in the Alerts pane.

Screen Shot 2017-06-18 at 15.29.51.png
The Alerts Pane

Alerts can be ignored, mark as read, previewed and imported. When an alert is imported, it becomes a case that needs to be investigated.

Cases

workflow
The Workflow that is at the Heart of TheHive

A case can be generated from an alert or created from scratch. It is subdivided into tasks (think identification, containment, eradication, check proxy logs, and so on) and observables (IP addresses, hashes, email addresses, domain names, URLs…). When analysts are working on tasks, they add logs as they go. In TheHive’s terminology, logs are text entries which may contain attachments to help analysts record what they have been doing. Logs can be written using Markdown or a rich-text editor.

Case Templates

You don’t need to add the same tasks over and over when working on cases belonging to a given category (DDoS, Malspam, APT, …). You can create custom templates to which you add tasks as shown below. This is very useful when you are dealing with alerts so that when you import them, you can select which case template you’d like to apply and there you go!

Screen Shot 2017-06-16 at 10.26.22.png
A Sample Case Template
Observables

Observables can be tagged, flagged as IOCs, and analyzed. When the investigation is well in progress or completed, you may want to share the resulting IOCs or a subset of those with partners and peers. TheHive will support the ability to export that data to MISP in September 2017. Until then, you can still export your IOCs as text, CSV or as a MISP-compatible format that you can use to add them to your MISP instance using the freetext editor. TheHive can export IOCs/observables in protected (hxxps://www[.]somewhere[.]com/) or unprotected mode.

Every observable must have a TLP (Traffic Light Protocol) level. By default, any added observable is considered TLP:AMBER. Please note that the TLP is taken into account by some analyzers. Wait! Analyzers?

Cortex

Cortex is our standalone analysis engine and a perfect companion for TheHive and MISP. Analysts can use it to analyze observables using its Web UI, in which case they can be submitted only one at a time. The Web UI should really be limited to quick assessments of observables before creating a case in TheHive (or in an alternate SIRP). The power of Cortex really comes into play when you use its REST API. TheHive speaks natively to Cortex (as MISP does). Moreover, TheHive can leverage one or several Cortex servers.

Screen Shot 2017-06-18 at 15.57.27.png
Observable Page and List of Analyzers
Analyzers

As of this writing, Cortex has 23 analyzers which come in a total of 39 flavors and more will be available soon.

An analyzer can be written in any programming language supported by Linux though all of our current analyzers are written in Python. This is because we provide a Python library called Cortexutils which contains a set of utility classes that make it easier to write an analyzer in Python.

Flavors

Analyzers such as VirusTotal, PassiveTotal or DomainTools can provide different analysis services. Let’s take VirusTotal as an example. You can scan a file or URL. That’s one flavor. You can also obtain the latest available report on VirusTotal.com for a file, hash, domain or IP address. That’s a second flavor. So the VirusTotal analyzer has two flavors.

Screen Shot 2017-06-18 at 16.26.41.png

How about PassiveTotal? It has 8 flavors: unique resolutions lookup, SSL certificate history lookup, malware lookup, passive DNS lookup, data enrichment lookup, SSL certificate details lookup, OSINT lookup and WHOIS data lookup.

The MISP Search Analyzer

At this point, we need to mention a special analyzer that may create some confusion if not understood correctly: the MISP Search analyzer. Thanks to it, Cortex has the ability to search observables within a MISP instance as represented by the arrow that goes from the Analyzers to MISP.

Screen_Shot_2017-06-19_at_08_03_54.png
Search for MISP Events Containing a Given Observable

When an observable is found in an event, Cortex will return the number of records found (i.e. the number of events where the observable has been found) and a list of links to those events with additional data.

Screen_Shot_2017-06-19_at_08_13_16.png
Searching for a Hash Using the MISP Search Analyzer from the Cortex Web UI
Screen Shot 2017-06-19 at 08.17.04.png
The Same Search Conducted from TheHive: Long Report
Screen Shot 2017-06-19 at 08.18.58.png
Mini-Report

The current version of the MISP Search analyzer can only search within a single MISP instance but in the near future, it will be able to support multiple ones.

MISP Expansion Modules

Besides its own analyzers (which include MISP Search described above), Cortex can also invoke MISP expansion modules. These are normally used by MISP to enrich attributes within events but Cortex can also take advantage of them to analyze observables.

There is some overlap between the native Cortex analyzers and MISP expansion modules. When choosing between a native analyzer or an expansion module, we highly recommend you select the former. The expansion modules are deactivated in the default Cortex configuration.

Jobs

When you submit an observable for analysis, Cortex will create a job and, if successful, it will generate an analysis report in JSON format. TheHive has the ability to parse those results and present them in a human-friendly fashion thanks to report templates we offer for free. So when you’ll submit an observable to Cortex from TheHive, you’ll get back a short (or mini) report and a long one. The first can be thought of as a really tiny Exec Analyst Summary while the second provides more insight and details.

Calling Cortex from MISP

In addition to the expansion modules we have just mentioned, MISP 2.4.73 and up can enrich attributes using Cortex analyzers. The configuration is pretty straightforward. So if all you are concerned about is threat intelligence and sharing, you may augment your visibility into a given threat represented as a MISP event by leveraging all current 23 Cortex analyzers and any future ones.

Conclusion

TheHive, Cortex and MISP are three open source and free products that can highly aid you combat threats and keep the ‘monsters’ at bay.

TheHive, as a SIRP, allows you to investigate security incident swiftly in a collaborative manner. Several analysts can work simultaneously on tasks & cases . While cases can be created from scratch, TheHive can receive alerts from different sources thanks to alert feeders which consume security events generated by multiple sources and feed them into TheHive using TheHive4py Python library. TheHive can also sync to one or several MISP instances to receive new and updated events which will appear in the alert pane with all the other alerts generated by other sources. Analysts can then preview new alerts to decide whether they need to be acted upon. If so, they can transform them into investigation cases using templates.

To analyze the observables collected in the course of an investigation and/or imported from a MISP event, TheHive can rely on one or several Cortex analysis engines. Cortex is another standalone product that we have developed which sole purpose is to allow you to analyze observables at scale thanks to its large number of analyzers, MISP expansion modules and any analyzer you might have developed on the side. Cortex has a REST API that can be used to empower other security products such as  ‘analytics’ software, alternate SIRPs or MISP.

The highly popular threat sharing platform can indeed enrich attributes thanks to Cortex as it has a native integration with it. And in a few months, you will also be able to export cases from TheHive as MISP events that you can share with peers and partners.

If you do share, you do care about our collective mission to defend the  digital assets that are under our watch from harm. So let us fight together as one.

 

 

Cortexutils 1.1.1 is Here

Lo and behold, The Chefs behind TheHive Project have been pretty busy at the code kitchen, continuously improving some of your favorite recipes. After releasing Mellifera 11.3 and Cortex 1.1.3, here comes the turn of Cortexutils, the Python library containing utility classes for Cortex analyzers.

Version 1.1.1 of the library solves an encoding issue, in fact a regression introduced by the previous release. It also corrects a situation where error reports wouldn’t be generated in case the analyzer did not require configuration (we have a handful of such creatures in store).

You can grab the new Cortexutils release through PIP. To update your existing installation, please run the following command:

 sudo pip install cortexutils --upgrade

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Cortex Passes the 20 Analyzers Mark

Thanks to the invaluable contributions of our growing and thriving user community, Cortex has now 6 more analyzers, bringing the total to 21. The new analyzers, released under our usual AGPL v3 license, are:

  • CIRCLPassiveDNS
  • CIRCLPassiveSSL
  • GoogleSafebrowsing
  • Nessus
  • Virusshare
  • Yara

All but one have been submitted by Nils Kuhnert of CERT-Bund. The Nessus analyzer has been contributed by our long-time friend Guillaume Rousse.

Cortexutils 1.1.0

While reviewing the submissions, we realized that a new version of the Cortexutils library was needed in order to support both Python 2 and 3. Hence we released version 1.1.0. You can grab it through PIP. To update your existing installation, please run the following command:

 sudo pip install cortexutils --upgrade

Note that Cortexutils 1.1.0 is required to benefit from these analyzers and future ones. If you are performing a fresh Cortex installation, follow the guide.

Installation

To install the new analyzers, grab the Cortex-Analyzers repository and unpack its content (or git pull the master  branch) in your existing /path/to/cortex-analyzers. Then follow the Cortex analyzers guide.

New Short and Long Report Templates for TheHive

Short and long reports for TheHive were also created to parse and display the results produced by the new analyzers. We also bundled in the new package a URL analysis template for Joe Sandbox which was missing and improved some of the older short templates in order to follow a taxonomy.

To import the new report templates in your instance of TheHive:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu
  • click on Import templates button and select the downloaded package

CIRCLPassiveDNS

The CIRCLPassiveDNS analyzer lets you check the CIRCL’s Passive DNS service for a given domain. It takes domains and URLs as input. Access to the service is allowed to trusted partners in Luxembourg and abroad. If you think you qualify, please contact the good CIRCL folks. To make it work, you’ll need the pypdns Python library.

In order to take advantage of CIRCLPassiveDNS, you need to add the following section to the Cortex configuration file (application.conf):

CIRCLPassiveDNS {
     user=""
     password=""
}

When called from TheHive, the following output is produced:

sc-short-CIRCLPassiveDNS.png

sc-long-CIRCLPassiveDNS.png
TheHive: CIRCLPassiveDNS Analyzer – Short and Long Report Samples

CIRCLPassiveSSL

The CIRCLPassiveSSL analyzer lets you check CIRCL’s Passive SSL service for a given IP address or certificate hash. Access to the service is restricted to partners and security researchers worldwide. If you think you qualify, please contact the good CIRCL folks. This analyzer needs the pypssl Python library to work properly.

To use it, please add the following section to the Cortex configuration file (application.conf):

CIRCLPassiveSSL {
     user=""
     password=""
}

When called from TheHive, the following output is produced:

sc-short-CIRCLPassiveSSL.png

sc-long-CIRCLPassiveSSL.png
TheHive: CIRCLPassiveSSL Analyzer – Short and Long Report Samples

GoogleSafebrowsing

This analyzer lets you check URLs and domains against Google Safe Browsing. You need an API key to use it.

To leverage GoogleSafebrowsing, add the following section to Cortex’ configuration file:

GoogleSafebrowsing {
  key="" 
}

When you run the analyzer fromTheHive, you should see output similar to the samples below:

sc-short-safebrowsing.png

sc-long-safebrowsing.png
TheHive: GoogleSafebrowsing Analyzer — Short and Long Report Samples

Nessus

The Nessus analyzer lets you leverage Tenable’s Nessus Vulnerability Scanner to scan an IP address or a FQDN. Of course, you must not scan assets that do not belong to you, unless you really know what you are doing. That’s why safeguards were built in the analyzer’s configuration:

Nessus {
   url="<https://url.to.scanner>"
   login=""
   password=""
   policy=""
   ca_bundle=""
   allowed_networks=[ 'x.y.z.t/8', 'a.b.c.d/24', ... ]

The nessrest Python library is needed to make REST API calls to Nessus. Analysts would use the analyzer to assess the vulnerabilities of potentially compromised machines or new, unknown assets that have been plugged into one of their constituency’s networks. Of course, penetration testers conducting large-scale reconnaissance can also benefit from this analyzer.

sc-short-nessus.png

sc-long-nessus.png
TheHive: Nessus Analyzer — Short and Long Report Samples

Virusshare

The Virusshare analyzer lets you verify whether a file or hash is available on VirusShare.com. It requires the progressbar2 Python library besides requests (which should be already installed if you have an existing Cortex installation). As stated by Nils:

This analyzer enables searching for md5 hashes in Virusshare.com hash list. It does not download samples for you nor links directly to the sample – the author of virusshare prohibits the automatic download/site scraping and I respect that. It provides a button to start the virusshare search, though, but you need an account for that. You can request an invitation to the platform through contacting the admin via mail, directly.

To use it, add the following section to your Cortex application.conf:

Virusshare {
   path="/path/to/download/directory"
}

Quoting Nils again, in order to download the newest available hash lists from virusshare.com, you can run the download_hashes.py script that comes with the analyzer.

./download_hashes.py /path/to/your/download/directory

Upon running the analyzer from TheHive, the report will contain a link to the corresponding Virusshare page if a match is found as shown below.

sc-long-virusshare.png
TheHive: Virusshare Analyzer — Long Report Sample

Yara

Last but not least, the Yara analyzer can check files against YARA rules using yara-python. To use it, add the following to your Cortex configuration file:

Yara {
    rules=["/path/a", "/path/b", "/path/my/rules.yar"]
}

You can specify path to directories and files. If you supply a directory, the analyzer expects to find an index.yar or index.yas file. The index file can include other rule files. An example can be found in the Yara-rules repository.

sc-short-yara.png

sc-long-yara.png
TheHive: Yara Analyzer — Short and Long Report Samples

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!