Cortex-Analyzers 2.4.0: 138 Ways to Analyze, 10 Methods to Respond

Guess what? TheHive Project is still alive and well, as Saâd already mentioned in a previous blog post.

We’ve been certainly very busy lately, preparing the upcoming release of TheHive 4 and doing many other things beside working on our FOSS project. As a result, it took us a rather long time to merge several community contributions and reduce the sizeable pile of pull requests.

We would like to thank our contributors for their patience and we hope the cyberdefenders out there will enjoy the brand new Cortex-Analyzers 2.4.0 release, with many new analyzers, responders and some bug fixes & improvements, bringing the total to a whooping 138 analyzers (counting all flavors) and 10 responders!

Additionally, with this release, all analyzers are now using Python 3. No more Python 2 technodebt!

Photo by Saâd Kadhi

What’s New?

New Analyzers

8 new analyzers have been added to this release:

1 analyzer has new flavors:

New Responders

3 new responders have been added:

Overview of the New Analyzers

DomainToolsIris

This analyzer looks up domain names, IP addresses, e-mail addresses, and SSL hashes using the popular DomainTools Iris service API.

The analyzer comes in 2 flavors:

  • DomainToolsIris_Investigate: use DomainTools Iris API to investigate a domain.
  • DomainToolsIris_Pivot: use DomainTools Iris API to pivot on ssl_hash, ip, or email.

A valid DomainTools API integration subscription is needed to run this analyzer.

TheHive displays the analyzer results as follows:

DomainToolsIris short report
DomainToolsIris long report

EmailRep

The EmailRep analyzer checks the reputation of an email address against the emailrep.io database.

IPInfo

This analyzer accesses IP-centric features provided by ipinfo.io. While the EmailRep API can be used without a token for limited usage, the ipinfo.io analyzer requires the configuration of an API token before use.

Maltiverse

This analyzer lets you query the free Maltiverse Threat Intelligence platform for enrichment information about a particular hash, domain, ip or url.

TheHive displays the analyzer results as follows:

Maltiverse short report
Maltiverse long report

MalwareClustering

Andrea Garavaglia contributed this one a long time ago and we finally merged it into the Cortex-Analyzers repository. Andrea gave a talk about the background of this analyzer at the fourth MISP summit. You can watch it here.

In order to use the analyzer, you need to point it to a Neo4j server (you need to supply the host, port, login & password).

PaloAlto Autofocus

This analyzer lets you leverage PaloAlto Autofocus services. Provided you are an Autofocus customer and you have access to their API, you need to configure the analyzer with your username and a token key.

The analyzer comes with 3 flavors:

  • AUTOFOCUS_GetSampleAnalysis lets you request a full report for a given hash.
  • AUTOFOCUS_SearchIOC lets you research for samples linked to specific IoCs with datatypes like domain, fqdn, user-agent, imphash, ip, mutex, tag and url. Please note that mutex and tag are not default datatypes in TheHive. You need to create them in TheHive before you can leverage them.
  • AUTOFOCUS_SearchJSON lets you research for samples based on a complex JSON query.

Important: TheHive has no templates corresponding to this analyzer have been published yet. They will be provided in the near future.

SpamhausDBL

This analyzer performs reputation lookups of a domain or a fqdn against Spamhaus Domain Block List (DBL).

TheHive displays the analyzer results as follows:

SpamhausDBL short report
SpamhausDBL long report

TeamCymruMHR

This analyzer queries Team Cymru’s Malware Hash Registry for known malware hashes (MD5 or SHA-1). If it is malware and known by the service, it returns the last time it has been seen along with an approximate anti-virus detection percentage.

Overview of the New Responders

KnowBe4

This responder allows the integration between TheHive/Cortex and KnowBe4’s User Events API.
If a mail observable is tagged with a specified tag, corresponding to the responder’s configuration (e.g. phished), then the associated user will have a custom event added to their profile in KnowBe4.

A valid account on KnowBe4 and an API key are required to run this responder.

Minemeld

This responder sends observables you select to a Palo Alto Minemeld instance.

To run this responder, a MineMeld Threat Intelligence Sharing account is needed.

Wazuh

This responder performs actions on Wazuh, the open source security monitoring platform. It currently supports ad-hoc firewall blocking of ip observables.

Improvements

New PassiveTotal flavors

Thanks to Brandon Dixon, the PassiveTotal analyzer gains 3 new flavors, bringing the total to 11:

  • PassiveTotal_Trackers let you make tracker lookups on observables of type domain, fqdn and ip.
  • PassiveTotal_Host_Pairs let you make host pair lookups on observables of type domain, fqdn and ip.
  • PassiveTotal_Components lets you make components lookup on observables of type domain, fqdn and ip.

They come with their own report templates.

GreyNoise Analyzer

The analyzer has been updated to support GreyNoise API v2, thanks to the contribution of Whitney Champion (#562).

New Data Types Supported by Some Analyzers

  • VirusTotal_GetReporthas been updated to allow requests for observables of type fqdn.
  • Threatcrowd has been updated to allow requests for observables of type domain.
  • Shodan has been updated to allow requests for observables of type fqdn.

Fixes

  • [#602] The MISP analyzer was bumped to version 2.1 and is ready to use PyMISP 2.4.120.

Get It While Supply Lasts!

I’m Hype

If you are using the dockerized analyzers & responders, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button.

I’m Country

If you are still using the old-style way of installing analyzers and responders, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Unveiling Cortex 2

TheHive Project’s Master Chefs are extremely happy to share, for free their latest recipe with the Cyber Threat Intelligence, Digital Forensics and Incident Response communities: Cortex 2.

As its predecessor, Cortex 2 is published under an AGPL v3 license and it introduces many important features that we brushed upon in a Dec 2017 post.

Screen Shot 2017-12-15 at 17.16.06
Cortex 2 — Architecture

Update: Cortex 2.0.1 was released since this post went live. It corrects a few bugs we uncovered in 2.0.0 as described in the changelog. Please install Cortex 2.0.1 instead of 2.0.0.

Authentication

Cortex 2 supports all the authentication methods that TheHive supports: LDAP, Active Directory, local accounts, API Keys, and X.509 SSO.

To connect your favorite Security Incident Response Platform with Cortex 2, you will need to update TheHive to Cerana 0.7 (TheHive 3.0.7) which was released today as well. This version fixes a regression pertaining to case templates introduced by Cerana 0.6 and is the first version to fully support Cortex 2’s API changes and authentication.

To make TheHive 3.0.7 analyze observables at scale through Cortex 2, you have to create an account on Cortex 2 with the read and analyze roles (see the next section) and generate the associated API Key. Next, feed the key in TheHive’s /etc/thehive/application.conf as described in the documentation et voilà !

TheHive 3.0.7 remains compatible with Cortex 1 and you can connect it to a mixed set of Cortex 1 and/or Cortex 2 instances with no issues.

Organizations, Analyzers and Rate Limiting

Cortex 2 introduces multi-tenancy through organizations and each organization can have its own set of users, with different roles, its own set of analyzers and, if necessary, rate limits that will prevent analysts from burning quotas.

Multi-tenancy has several interesting use cases. For instance, if you are the CSIRT or CERT of a large multinational organization with several regional teams, you can create an organization for each region within your constituency and enable the analyzers that they may need to use. Let’s assume that you bought a VirusTotal subscription that limits you to 5000 requests per month. You can configure the corresponding analyzers to give each region a fair share of that quota and keeping some requests for your own use.

In case you are a commercial CSIRT or an MSSP, you could do the same for your customers by installing only one Cortex 2 instance and creating an organization for each customer.

Screen Shot 2018-03-29 at 16.27.05.png
Configure an analyzer graphically and impose rate limits if necessary

User Roles

By default, Cortex 2 is shipped with the default cortex organization which sole purpose is to create other ones and manage the users within each organization and their associated powers. The cortex organization hosts all users with the superAdmin role and it cannot be used to configure or run analyzers.

As described in the new Quick Start Guide, after installing Cortex 2, updating its database and creating the first user who will have super admin powers, you’ll have to create your first organization and at least one user within that organization with orgAdmin rights.

Screen Shot 2018-03-29 at 16.33.02
Create an organization

You can then log out and log in using the orgAdmin account to create further users within that organization, enable and configure analyzers etc. Please note that no analyzer is enabled by default and you need at least v 1.9.0 of the cortex-analyzers repository. To update your set of analyzers to 1.9.0, please run git pull.

Screen Shot 2018-03-29 at 16.28.47
Manage users within an organization

Besides the superAdmin and orgAdmin roles, Cortex 2 introduces the read role which allows users to access analyzer reports and read them but not execute analyzers. For that, users need the analyze role (which implies the read role). orgAdmin users can also run analyzers. superAdmin users are limited to the default cortex organization. While they can create organizations and manage users within them, they cannot access analyzer configurations such as confidential API keys or job reports.

Screen Shot 2018-03-29 at 16.31.28
Job reports

Report Persistence and Caching

Cortex 2 relies on Elasticsearch 5.x to store many configuration items but also all the analyzer reports that have been generated. Unlike its predecessor, you won’t lose your existing reports should you need to restart the service or the host it is running on.

Cortex 2 also introduces report caching. By default the cache.job parameter is set to 10 minutes in /etc/cortex/application.conf. That means that if an analysis on a given observable with a defined TLP is requested and that a report has been previously generated in the last 10 minutes, Cortex 2 will serve that report instead of running a new analysis. This feature can help prevent soliciting analyzers, particularly those which require a subscription or have quotas, when there is no need to do so. Please note that this parameter is global to all the analyzers and all the organizations that are configured in the Cortex 2 instance. We do have plans to make it more granular in future versions.

Migrating from Cortex 1

If you are migrating from Cortex 1.x, we recommend that you:

  1. Save the configuration of your analyzers (which ones are enabled and what their configuration items are, such as users/passwords or API keys).
  2. Install Cortex 2.
  3. Edit /etc/cortex/application.conf to add the secret key as shown in Step 1 of the Quick Start Guide and point Cortex to the location of the analyzers.
  4. Follow the remaining steps of the Quick Start Guide to enable the analyzers you need and reinject their configuration.

What’s Next?

In the upcoming weeks, we will release a new version of Cortex4py in order to make it compatible with Cortex 2, continue the work we started with our MISP Project friends to support MISP attribute enrichment through Cortex 2 (MISP currently only supports enrichment using Cortex 1), and perform a long-overdue overhaul of our documentation.

Feeling Generous? Donate!

As you know, we are a FOSS project and donations are always welcome to make our products even better for the community.

All donations go to Creative Source, the non-profit organization we have created, and we will use them to improve TheHive, Cortex & Hippocampe but also to develop (even better) integrations with other FOSS solutions such as MISP.

So if you are feeling generous, please contact us at support@thehive-project.org.

Creative Source can also provide so-called professional, entreprise-grade support, help integrating the products, train your analysts before they drain or assist you in specific areas such as developing in-house analyzers for Cortex.

Support

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

Correction: March 30, 2018
Instructions on how to update the cortex-analyzers have been added. Also, Cortex 2.0.1 was released to correct a few bugs in the previous version since this post went live.

WOT? Did You See a Yeti Hugging a Cuckoo?

While many are enjoying the summer holidays, the busy bees of TheHive Project have been working hard lately to develop new Cortex analyzers and review few of those submitted by our growing and thriving user community, bringing the grand total to 27. Yes, you read that right. Cortex can leverage 27 analyzers to help you analyze observables very simply in many different ways.

The latest update to the Cortex-analyzers repository contains 3 new analyzers: Yeti, Cuckoo Sandbox and WOT, described below. And your first step to benefit from them should consist of refreshing your master working copy on your Cortex instance:

$ cd where/your/analyzers/are
$ git pull master

Yeti

YETI is a FOSS platform meant to organize observables, indicators of compromise, TTPs, and knowledge on threats in a single, unified repository.  It is mainly developed by fellow APT busters Thomas Chopitea and Gael Muller (who said France doesn’t produce good software?).

The new Cortex analyzer for this platform lets you make API calls to YETI and retrieve all available information pertaining to a domain, a fully qualified domain name, an IP address, a URL or a hash.

To be able to use the analyzer edit the Cortex configuration file (/etc/cortex/application.conf) and add the following lines:

Yeti {
    # URL of the Yeti server: example: http://120.0.0.1:5000
    url = ""
}

When called from TheHive, the following output is produced:

sc-YETI-short.png

sc-YETI-long.png
TheHive: YETI analyzer — Short and Long Report Samples

CuckooSandox

The Cuckoo Sandbox analyzer has been submitted by Andrea Garavaglia (Thanks!) and you can use it to analyze files and URLs with Cuckoo Sandbox.

By default, we chose to limit analysis to TLP:WHITE and TLP:GREEN observables for OPSEC reasons, in case your Cuckoo server provides Internet access to potentially harmful files. If you want to use it with TLP:AMBER or TLP:RED observables, edit CuckooSanbox_File_analysis.json or CuckooSanbox_URL_analysis.json and change the max_tlp parameter to 2 or 3.

To use the analyzer, edit the Cortex configuration file and add the following lines:

CuckooSandbox {
   url = “http://mycuckoosandbox”
}

When called from TheHive, the following output is produced:

sc-CSB-short.png

sc-CSB-long.png
TheHive: Cuckoo Sandbox Analyzer — Short and Long Report Samples

WOT

The WOT analyzer was also submitted by Andrea Garavaglia (kudos!). Use it to check reputation of a given domain on the Web of Trust service. It takes domains and FQDNs as input.

An API key is needed to use this service, and has to be added in the Cortex configuration file:

WOT {
    # API key of the Web of Trust account
    key=“”
}

When called from TheHive, the following output is produced:

sc-WOT-short.png.png

sc-WOT-long.png.png
TheHive: WOT Analyzer — Short and Long Report Samples

Support

Something does not work as expected? No worries, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.