Cortex4py 2 is Out!

Cortex, a free, open source software allows security analysts and threat hunters to analyze and enrich observables (IP addresses, hashes, domains, …) collected in the course of an investigation or received from third parties, for example through MISP, the de facto standard for threat sharing.

On March 29, 2018, we released Cortex 2, a major improvement over the previous version which brought, among other cool features, authentication, caching, multi-tenancy (RBAC) and rate limiting. Instead of deploying several Cortex 1 instances behind reverse proxies which would implement authentification, administrators can deploy a single Cortex 2, create multiple organizations and serve the needs of various information security populations while enjoying extra features.

On May 31, 2018, we published a brand new API guide so that developers can take advantage of the powerful REST API of the product. Sadly, Cortex4py, the FOSS Python library we provide to interact with the API was not compatible with Cortex 2. Until today.

Thanks to the hard work of our dear Nabil Adouani, we are happy to announce the immediate availability of Cortex4py 2.0.0, a complete rewrite of the library in Python 3. Cortex4py 2.0.0 is fully compatible with Cortex 2. However, it doesn’t work with Cortex 1.

While TheHive, the highly popular free and open source Security Incident Response Platform (SIRP) we develop has native support for many Cortex 2 instances, Python developers can leverage Cortex4py to interact with Cortex 2, manage organizations, users, analyzer configurations and analyze observables at scale from alternative SIRPs, SIEMs or custom scripts thanks to the 83 analyzers Cortex 2 has as of June 18, 2018.

Screen Shot 2018-06-18 at 20.01.27.png
Cortex 2: there is more than one way to interact with it

Use It

To install Cortex4py, use PIP3:

$ sudo -H pip3 install cortex4py

If you are using Python on a Windows operating system, please forgo the sudo command.

Usage

Cortex4py 2 comes with a usage guide which includes many examples. For example, if you want to fetch the last 10 successful jobs that have been executed against domain names and display the result summaries of those 10 jobs you could write something like:

Screen Shot 2018-06-18 at 19.58.45.png
Sample Python3 code to retrieve Cortex analyzer results

Migrating from Cortex4py 1

If you have already written scripts using Cortex4py 1.x (for Cortex 1), we tried to keep the already available methods. However, we recommend you adapt your code to leverage the new Cortex4py 2 classes and methods as soon as feasible. Moreover, the existing scripts must be updated to support authentication if you intend to use them with Cortex 2. Please read the Cortex4py 2 usage guide for more information.

Support

Cortex 2.0.0 is brand new software. As such, it might contain bugs and limitations. If you find any or encounter problems, please ask on our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

Cortex 2, TheHive and a Whole Slew of Updates

After announcing Cortex 2.0.0 and TheHive 3.0.7, the first version of your favorite SIRP that is (supposedly) compatible with the brand-new version of Cortex, last week, we thought it was time to relax and enjoy the upcoming, long Easter weekend, the sunny sky of Paris (if you can pierce the veil of the Forever Grey Cloud™ that is hanging over the city of lights), and great jazz music. Heck, I even tweeted about it … only to be proven wrong by Life (and Murphy).

We literally field tested Cortex 2 for 3 weeks, we squashed bugs here and there, until almost the very last minute before the release. And yet, our QA needs to be improved by leaps and bounds as we had to release Cortex 2.0.1 one day after unveiling 2.0.0 to correct some additional bugs. And then some members of the core team and of our growing user community took it for a spin. And all hell broke lose. Well, almost 🙂

good_code
Source: XKCD

Session collisions (when TheHive and Cortex 2 are used on the same machine), analyzer malfunctions, connectivity problems … issues that were not identified during the testing phase, even in a production environment, where everything worked as expected. And we call this ‘Computer Science’. Right, right…

So we worked hard, took out our Code Hammer (it’s like Thor’s but cyber) and blasted away all the bugs that we found out or that were reported to us (arigato gozaimasu!) and we are happy to announce the immediate availability of Cortex 2.0.2, TheHive 3.0.8, Cortexutils 1.2.3 and Cortex-Analyzers 1.9.2.

TL;DR Install or upgrade Cortex 2.0.2, update Cortexutils, git pull the Cortex-analyzers repo to get the latest version of the repository, upgrade to TheHive 3.0.8, follow the Quick Start Guide and have a drink.

If you have time (which is admittedly quite scarce nowadays), please read on the changelogs:

What’s Next?

As stated in the previous post, we will release a new version of Cortex4py in order to make it compatible with Cortex 2, continue the work we started with our MISP Project friends to support MISP attribute enrichment through Cortex 2 (MISP currently only supports enrichment using Cortex 1), and perform a long-overdue overhaul of our documentation. We will also release a brand new version of TheHive4py.

Last but not least, we’ll take a hard look at ourselves and our QA. You expect us from us high quality and we hold ourselves to high standards. And we will deliver.

Support

Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

Unveiling Cortex 2

TheHive Project’s Master Chefs are extremely happy to share, for free their latest recipe with the Cyber Threat Intelligence, Digital Forensics and Incident Response communities: Cortex 2.

As its predecessor, Cortex 2 is published under an AGPL v3 license and it introduces many important features that we brushed upon in a Dec 2017 post.

Screen Shot 2017-12-15 at 17.16.06
Cortex 2 — Architecture

Update: Cortex 2.0.1 was released since this post went live. It corrects a few bugs we uncovered in 2.0.0 as described in the changelog. Please install Cortex 2.0.1 instead of 2.0.0.

Authentication

Cortex 2 supports all the authentication methods that TheHive supports: LDAP, Active Directory, local accounts, API Keys, and X.509 SSO.

To connect your favorite Security Incident Response Platform with Cortex 2, you will need to update TheHive to Cerana 0.7 (TheHive 3.0.7) which was released today as well. This version fixes a regression pertaining to case templates introduced by Cerana 0.6 and is the first version to fully support Cortex 2’s API changes and authentication.

To make TheHive 3.0.7 analyze observables at scale through Cortex 2, you have to create an account on Cortex 2 with the read and analyze roles (see the next section) and generate the associated API Key. Next, feed the key in TheHive’s /etc/thehive/application.conf as described in the documentation et voilà !

TheHive 3.0.7 remains compatible with Cortex 1 and you can connect it to a mixed set of Cortex 1 and/or Cortex 2 instances with no issues.

Organizations, Analyzers and Rate Limiting

Cortex 2 introduces multi-tenancy through organizations and each organization can have its own set of users, with different roles, its own set of analyzers and, if necessary, rate limits that will prevent analysts from burning quotas.

Multi-tenancy has several interesting use cases. For instance, if you are the CSIRT or CERT of a large multinational organization with several regional teams, you can create an organization for each region within your constituency and enable the analyzers that they may need to use. Let’s assume that you bought a VirusTotal subscription that limits you to 5000 requests per month. You can configure the corresponding analyzers to give each region a fair share of that quota and keeping some requests for your own use.

In case you are a commercial CSIRT or an MSSP, you could do the same for your customers by installing only one Cortex 2 instance and creating an organization for each customer.

Screen Shot 2018-03-29 at 16.27.05.png
Configure an analyzer graphically and impose rate limits if necessary

User Roles

By default, Cortex 2 is shipped with the default cortex organization which sole purpose is to create other ones and manage the users within each organization and their associated powers. The cortex organization hosts all users with the superAdmin role and it cannot be used to configure or run analyzers.

As described in the new Quick Start Guide, after installing Cortex 2, updating its database and creating the first user who will have super admin powers, you’ll have to create your first organization and at least one user within that organization with orgAdmin rights.

Screen Shot 2018-03-29 at 16.33.02
Create an organization

You can then log out and log in using the orgAdmin account to create further users within that organization, enable and configure analyzers etc. Please note that no analyzer is enabled by default and you need at least v 1.9.0 of the cortex-analyzers repository. To update your set of analyzers to 1.9.0, please run git pull.

Screen Shot 2018-03-29 at 16.28.47
Manage users within an organization

Besides the superAdmin and orgAdmin roles, Cortex 2 introduces the read role which allows users to access analyzer reports and read them but not execute analyzers. For that, users need the analyze role (which implies the read role). orgAdmin users can also run analyzers. superAdmin users are limited to the default cortex organization. While they can create organizations and manage users within them, they cannot access analyzer configurations such as confidential API keys or job reports.

Screen Shot 2018-03-29 at 16.31.28
Job reports

Report Persistence and Caching

Cortex 2 relies on Elasticsearch 5.x to store many configuration items but also all the analyzer reports that have been generated. Unlike its predecessor, you won’t lose your existing reports should you need to restart the service or the host it is running on.

Cortex 2 also introduces report caching. By default the cache.job parameter is set to 10 minutes in /etc/cortex/application.conf. That means that if an analysis on a given observable with a defined TLP is requested and that a report has been previously generated in the last 10 minutes, Cortex 2 will serve that report instead of running a new analysis. This feature can help prevent soliciting analyzers, particularly those which require a subscription or have quotas, when there is no need to do so. Please note that this parameter is global to all the analyzers and all the organizations that are configured in the Cortex 2 instance. We do have plans to make it more granular in future versions.

Migrating from Cortex 1

If you are migrating from Cortex 1.x, we recommend that you:

  1. Save the configuration of your analyzers (which ones are enabled and what their configuration items are, such as users/passwords or API keys).
  2. Install Cortex 2.
  3. Edit /etc/cortex/application.conf to add the secret key as shown in Step 1 of the Quick Start Guide and point Cortex to the location of the analyzers.
  4. Follow the remaining steps of the Quick Start Guide to enable the analyzers you need and reinject their configuration.

What’s Next?

In the upcoming weeks, we will release a new version of Cortex4py in order to make it compatible with Cortex 2, continue the work we started with our MISP Project friends to support MISP attribute enrichment through Cortex 2 (MISP currently only supports enrichment using Cortex 1), and perform a long-overdue overhaul of our documentation.

Feeling Generous? Donate!

As you know, we are a FOSS project and donations are always welcome to make our products even better for the community.

All donations go to Creative Source, the non-profit organization we have created, and we will use them to improve TheHive, Cortex & Hippocampe but also to develop (even better) integrations with other FOSS solutions such as MISP.

So if you are feeling generous, please contact us at support@thehive-project.org.

Creative Source can also provide so-called professional, entreprise-grade support, help integrating the products, train your analysts before they drain or assist you in specific areas such as developing in-house analyzers for Cortex.

Support

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

Correction: March 30, 2018
Instructions on how to update the cortex-analyzers have been added. Also, Cortex 2.0.1 was released to correct a few bugs in the previous version since this post went live.

Privilege Escalation Vulnerability in All Versions of TheHive

Jeffrey Everling has identified a nasty privilege escalation vulnerability in all versions of TheHive, including Mellifera 13.2 (TheHive 2.13.2) and Cerana 0.2 (TheHive 3.0.2). Jeffrey reported it to us today Friday, Dec 22, 2017. Thanks but we could think of a better Christmas gift 😉

The vulnerability allows users with read-only or read/write access to escalate their privileges and eventually become administrators. To exploit it, an attacker must have access to an account on TheHive with read-only or read/write privileges.

The attacker needs to interact with the API in a specific though trivial way to obtain administrator privileges. After verifying that their request has been correctly processed, they connect to TheHive using the Web UI and they will see the administrator menu from where they can edit or lock user accounts, add case templates, etc.

We highly recommend you update TheHive to Cerana 0.3 (TheHive 3.0.3) which fixes the vulnerability. If you are still using Mellifera and have not made the move to Cerana yet, please update to Mellifera 13.3 (TheHive 2.13.3) which also corrects this flaw.

If you cannot immediately apply the hotfixes we have released, we have created a shell script that will allow you to spot anyone who exploited the vulnerability. You can download the script from the following location:

https://drive.google.com/file/d/1F8VOUMLoCVnIdHjnbhMTzf_9Z2Ud_Vuw/view?usp=sharing

The SHA256 hash of the script is:

18c74f921b92cc68ea7bc10c7522691d671074331191fe22269cc936bfdb0e9a

When you run the script, it will display all users that have changed their roles. If single match is found, it means your  instance  has  been  potentially compromised. We advise you to create a crontab which will execute the script on a regular basis until you apply the hotfixes.

To Upgrade to Cerana 0.3 (TheHive 3.0.3)

Start by following the migration guide.

If you are performing a fresh installation, read the installation guide corresponding to your needs and enjoy. Please note that you can install TheHive using an RPM or DEB package, use Docker, install it from a binary or build it from sources.

To Upgrade to Mellifera 13.3 (TheHive 2.13.3)

DEB Package

wget https://dl.bintray.com/cert-bdf/debian/TheHive_2.13.3-1_all.deb​​​​​dpkg -i TheHive_2.13.3-1_all.deb

The SHA256 hash of the DEB package is:

68c606fb9cbd56f63ba1f2d29c7f7652f4848c7783a6da574532bed0c963829b

RPM Package

wget https://dl.bintray.com/cert-bdf/rpm/thehive-2.13.3-1.noarch.rpm
rpm -Uvh thehive-2.13.3-1.noarch.rpm

The SHA256 hash of the RPM package is:

e566418bf861b2bf28842cf92f5c5d475c98fee1a3ae0d65e3990fd061a0bce0

Docker

docker run certbdf/thehive:2.13.3-1

Binary Package

wget https://dl.bintray.com/cert-bdf/thehive/thehive-2.13.3.zip

The SHA256 hash of the binary package is:

54c589f929744096b50d01264b9d4cc8b9e3d30d397fe810879b4d16b81287c1

Unzip the file in the folder of your choosing.

Support

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

TheHive4py 1.4.0 Released

Version 1.4.0 of the Python API client for TheHive is now available. It is compatible with the freshly released Cerana (TheHive 3.0.0).

We’d like to thank Nick Pratley, a frequent contributor, Bill Murrin, Alexander Gödeke and “srilumpa” for their code additions and documentation.

To update your existing package:

$ sudo pip install thehive4py --upgrade

If you are just getting started with TheHive4py, you can forgo the --upgrade at the end of the command above.

New Features

  • #5: Add a method to update a case, contributed by Nick Pratley
  • #34: Add a get_task_logs method in order to obtain all the task logs associated with a given taskId. Contributed by Bill Murrin
  • #37: A new, very cool case helper class by Nick Pratley
  • #39: Add support for custom fields to the case model
  • #40: Ability to run a Cortex analyzer through the API by Alexander Gödeke
  • #45: Simplify case creation when using a template by providing just its name
  • #49: Add a query builder capability to support TheHive’s DSL query syntax

Paris? Are you There?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. As usual, we’ll be more than happy to help!

Introducing Cerana

Update: 2 days after publishing this blog post, we’ve released Cerana 0.1 (TheHive 3.0.1) which fixes a number of issues. We encourage you to use 3.0.1 instead of 3.0.0.

The friendly honeybees at TheHive’s code kitchen were pretty busy lately even though winter came and temperatures have been close to zero Celsius in Paris, France. As we wrote a couple of weeks ago on this very blog, we are happy to announce Cerana to the world, available immediately.

Cerana or TheHive 3.0.0 is the latest (and obviously greatest) release of a now highly popular open source, free Security Incident Response Platform (or SIRP for short). Its flagship feature in comparison to previous releases is Dynamic Dashboards.

Dynamic Dashboards

Dynamic Dashboards replace the Statistics module in Cerana to allow you to explore the data available in Elasticsearch, which TheHive uses for storage, in many ways. For example, you can have a usage breakdown of Cortex analyzers, the number of open cases per assignee, the number of alerts per source (MISP, email notifications, DigitalShadows, Zerofox, Splunk, …), the number of observables that have been flagged as IOCs in a given time period, how many attributes were imported from MISP instances, top 10 tags of imported MISP attributes or incident categories.

case3.png
Dynamic Dashboards

Dynamic Dashboards can be created by an analyst and kept private or shared with the other team members. Dashboards can also be exported and imported into another instance. This would facilitate community participation in the establishment of valuable data exploration graphs to drive DFIR activity and seek continuous improvement.

When you’ll migrate to Cerana, you won’t have to build dashboards from scratch. We recreated more or less those which were available under the Statistics view and included them in the Cerana build.

Cortex and MISP Health Status

Cerana will also allow you to monitor the health status of all the Cortex and MISP instances that it is connected to. In the bottom right corner of TheHive’s Web UI, the Cortex and MISP logos appear when you have configured the integration with those products as in previous releases. However, the logos will have a small outer circle which color will change depending on whether Cortex and/or MISP instances are reachable or not.

status
Cortex & MISP Health

If TheHive can’t reach N out of M Cortex/MISP instances, the outer circle will be orange. If it can’t reach all M instances, the circle will red. If everything is fine, the circle will be green. The exact status of each Cortex/MISP instance can be seen in the About page. And when you try to run analyzers on a Cortex which cannot be reached, TheHive will tell you so as well.

about
Cortex & MISP: Version & Status

Sighted IOCs

In previous releases of TheHive, observables can be flagged as IOCs. However, this doesn’t necessarily mean you’ve seen them in your network. Think for example of a suspicious attachment which you’ve submitted to Cuckoo or Joe Sandbox through Cortex. The analyzer returns some C2 addresses to which the sample tries to connect to. You’d be right to add those C2 addresses to your case and flag them as IOCs. Then you search for them in your proxy logs and you find connection attempts to one out of four. In previous versions, you’d add a seen label but this would be inconsistent among analysts. One may use found instead. Another will add a description and no labels.

To avoid such situations and give you a simple way to declare an IOC as seen, Cerana adds a sighted toggle which you can switch on/off. We will leverage this toggle in future versions to indicate sightings when sharing back cases to MISP.

Other Features and Improvements

Cerana contains numerous other features and improvements such as:

  • Case template import, export
  • The ability to assign default values to metrics and custom fields to case templates
  •  The ability to assign by default tasks to their rightful owners in case templates
  • Show already known observables when previewing MISP events in the Alerts page
  • Add autonomous systems to the list of default datatypes
  • Single-sign on using X.509 certificates (in BETA currently)

We will update the documentation for Cerana in the upcoming weeks. So stay tuned.

Download & Get Down to Work

If you have an existing installation of TheHive, please follow the migration guide.

If you are performing a fresh installation, read the installation guide corresponding to your needs and enjoy. Please note that you can install TheHive using an RPM or DEB package, use Docker, install it from a binary or build it from sources.

Support

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

Cortex Hits the 30 Analyzers Mark

Cortex has now 30 analyzers thanks to Daniil Yugoslavskiy, Davide Arcuri and Andrea Garavaglia (from LDO-CERT) as well as our longtime friend Sébastien Larinier. Their contributions, all under an AGPLv3 license, add handy ways to assess observables and obtain invaluable insight to an already solid Threat Intelligence and DFIR toolset.

In addition to these 3 new analyzers, v 1.7.0 of the Cortex-Analyzers repository also fixes a number of bugs and add a few improvements to existing analyzers as well.

To get the new release, go to your existing Cortex-Analyzers folder and run git pull.

HybridAnalysis

The HybridAnalysis analyzer has been contributed by Daniil Yugoslavskiy. It fetches Hybrid Analysis reports associated with hashes and filenames. This analyzer comes in only one flavor called HybridAnalysis_GetReport.

Requirements

You need to have or create a free Hybrid Analysis account.  Follow the instructions outlined on the Hybrid Analysis API page to generate an API key/secret pair. Provide the API key as a value for the key parameter and the secret as a value to the secret parameter, add the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.

HybridAnalysis {
  secret = "mysecret"
  key = "myAPIKEY"
}

When run from TheHive, the analyzer produces short and long reports such as the following:

sc-short-hybridanalysis_1_0.png

TheHive: HybridAnalysis 1.0 Analyzer – Short and Long Report Samples
TheHive: HybridAnalysis 1.0 Analyzer – Short and Long Report Samples

EmergingThreats

The EmergingThreats analyzer has been submitted by Davide Arcuri and Andrea Garavaglia  from LDO-CERT. It leverages Proofpoint’s Emerging Threats Intelligence service to assess the reputation of various observables and obtain additional and valuable information on malware.

The service comes in three flavors:

  • EmergingThreats_DomainInfo: retrieve ET reputation, related malware, and IDS requests for a given domain.
  • EmergingThreats_IPInfo: retrieve ET reputation, related malware, and IDS requests for a given IP address.
  • EmergingThreats_MalwareInfo: retrieve ET details and info related to a malware hash.

Requirements

You need a valid Proofpoint ET Intelligence subscription.  Retrieve the API key associated with your account and provide it as a value to the key parameter, add the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.

 EmergingThreats {
   key="MYETINTELKEYGOESHERE"
 }

When run from TheHive, it produces short and long reports such as the following:

sc-short-ET_1_0.png

sc-long-ET-1_1_0.png

sc-long-ET-2_1_0.png

sc-long-ET-3_1_0.png

sc-long-ET-4_1_0.png

sc-long-ET-5_1_0.png
TheHive: EmergingThreats 1.0 Analyzer – Short and Long Report Samples

Shodan

The Shodan analyzer is the first submission by Sébastien Larinier. It lets you retrieve key Shodan information on domains and IP addresses.

This analyzer comes in two flavors:

  • Shodan_Host: get Shodan information on a host.
  • Shodan_Search: get Shodan information on a domain.

Requirements

You need to create a Shodan account and retrieve the associated API Key. For
best results, it is advised to get a Membership level account, otherwise a free one can be used.

Supply the API key as the value for the key parameter, add the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.

Shodan {
  key= "myawesomeapikey"
}

When run from TheHive, it produces short and long reports such as the following:

sc-short-shodan_1_0.png

sc-long-shodan_1_0.png
TheHive: Shodan 1.0 Analyzer – Short and Long Report Samples

Miscellaneous Fixes and Improvements

  • #100 : support both Cuckoo versions – by Garavaglia Andrea
  • #113 : Cuckoo Analyzer requires final slash – by Garavaglia Andrea
  • #93 : VirusTotal URL Scan Bug
  • #101 : Missing olefile in MsgParser requirements
  • #126 : PhishTank analyzer doesn’t work – by Ilya Glotov

Update TheHive Report Templates

If you are using TheHive, get the last version of  the report templates and import them into TheHive.

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!