More than 120 ways of Analyzing your Observables

There’s a new version of Cortex-Analyzers in cybertown and its has an awesome, mind-blowing name and that’s… wait for it… wait for it: 2.1.0.

In this new release, we added two analyzers which bring the total number to more than 120:

We could not duly test DNSSinkhole since we do not have access to the associated service. So we would really appreciate it if you could test it and let us know whether it works or not.

Others analyzers have been fixed or improved:

DNSSinkhole

This analyzer lets you check if an IP address has been registered in your DNS sinkhole. TheHive displays the analyzer results as follows:

DNSSinkhole analyzer: long report
DNSSinkhole analyzer: short report

TalosReputation

This analyzer lets you determine whether an IP address has been reported as a threat on Cisco Talos Intelligence service. No special access to the service is required to run the analyzer.

TheHive displays the analyzer results as follows:

Talos Reputation: long report
Talos Reputation analyzer : short report

Crt.sh

This analyzer has been enriched to display SHA-1 fingerprints. The long report format has been updated to reflect this new information.

Crt.sh analyzer: long report

FileInfo

FileInfo has been updated and is now able to parse PDF files and extract IOCs such as URLs, hosts, domains, IPs, hashes and many more.The analyzer does also support the last version of the extract-msg library.

FileInfo analyzer: IOC Parser long report
FileInfo analyzer: IOC Parser short report

VirusTotal and Python3

The VirusTotal analyzer, including all its flavours, now uses Python3 and an updated virustotal-api library.

Yeti API key

An optional API key can now be configured and used by the Yeti analyzer.

Malwares_GetReport

A hash computation has been fixed in this analyzer.

EMLParser

A first fix has been introduced to avoid this analyzer to crash when there is no content-description in content_header, and a second has been added to correct a header display issue.

IBM XForce Lookup

The analyzer has been improved to allow users to add a trailing / at the end of the API URL without breaking everything.

Updating your Analyzers in Cortex 2.x

Each analyzer and responder comes with its own, pip compatible requirements.txt file. Run the following commands to update your Cortex analyzers to the latest version:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip2 install -U -r $I || true; done && \
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, do not forget to login to Cortex as an orgadmin and click on the Refresh Analyzers button. Refer to the online Cortex documentation for further details.

Dockerised Analyzers

Cortex 3.x gives you the opportunity to run dockerised analyzers and responders. This means that you no longer have to download all the git repository of Cortex-Analyzers and run lengthy commands to update your analyzers and responders.

If you want to use dockerised analyzers and responders, ensure that the URL of the catalog.json file corresponding to the Cortex-Analyzers repository is registered in application.conf. Please note that this won’t work if you are tracking the stable catalog.

After doing so, do not forget to login to Cortex as an orgadmin, click on the Refresh Analyzers button, then Disable and Enable again each analyzer and responder. Analyzer (and responder) updates should occur automatically as long as docker.autoUpdate is set to true in application.conf (this is the default setting).

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Searching for an Elastic? Here, Take 6!

As we announced on May 14, 2019, we have been working very hard to add Elasticsearch 6 support to TheHive and Cortex as Elasticsearch 5.x went the way of the dodo when Elastic plugged life support off this venerable version. We also took this occasion to upgrade AngularJS and its sub projects to 1.7.8, the latest 1.x version as of this writing. Additionally, Grunt build dependencies have also been updated to their latest compatible versions.

It took us more time than initially foreseen but hey, we all love deadlines. We all love the whooshing noise they make as they go by.

TheHive 3.4.0-RC1 and Cortex 3.0.0-RC3 are now available on every Internet pipe near you and before you take them for a spin to help us identify any issues to make the stable releases rock-solid, let us walk you through some important information. Relax and grab a drink (and send good wine our way, we can always use some!).

Source: https://dilbert.com/strip/1995-11-10

TheHive 3.4.0-RC1

In addition to ES5 and 6 support and the update of AngularJS, this version corrects a few bugs that were identified in the latest stable version (3.3.1) and adds a few features. The most important one in our opinion is the ability to import a file from a Cortex report. This requires Cortex 3.0.0-RC3. The full list of changes is available at the following location.

Prior to migrating to 3.4.0-RC1, please read the migration guide.

Cortex 3.0.0-RC3

ES5 and ES6 support, AngularJS et cetera et cetera. Well you know the song right? Not quite as Cortex 3.0.0 significantly facilitates analyzer and responder installation and updates, thanks to Docker as we touched upon in a blog post earlier this year.

As detailed in the Cortex migration guide, which we recommend you read thoroughly, you can migrate from Cortex 2 and keep using analyzers and responders the same way (using processes), use the new Docker-based analyzers and responders or mix and match between running processes and docker containers (but then, you gotta pay extra attention to configure properly which analyzer/responder runs in which fashion).

Moreover, if you use the new dockerised analyzers and responders, you will be able to choose if you want to have them autoupdated (that’s the default behaviour) and if so, pick the bleeding edge, potentially buggy versions, the minor releases or, if you are risk-averse, stick with stable ones.

Cortex 3.0.0-RC3 also adds the ability to retrieve files resulting from analyzer jobs and last but not least, corrects an information disclosure bug that allowed non-admin users to retrieve the details of other users through the API. The vulnerability was reported by Adam Maris so kudos to him!

Warning: Regressions Ahead!

As outlined in our previous post about these new versions:

  • TheHive 3.4.0-RC1 and Cortex 3.0.0-RC3 use HTTP transport (9200/tcp by default) to connect to Elasticsearch instead of its native binary protocol (9300/tcp by default).
  • SSL/TLS, including when using a client certificate, can be configured to connect securely to ES. However this has not been tested yet.
  • Support of X-Pack and Search Guard is discontinued for anything but basic and SSL client authentication, which would still work.

Caution: Performance May Take a Hit!

The parent-child relationships we use behind the scene in Elasticsearch could make queries significantly slower with ES 6 and in our limited testing, we had the impression that performance took a hit. So please be cautious there and we’d be grateful if you could report any sluggishness you notice during your tests of the new versions with ES6.

The Dockerization Will not be Televised

Do you know what the following set of commands achieve?

$ cd /opt/Cortex-Analyzers
$ sudo git pull
$ for I in $(find /opt/Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip2 install -U -r $I; done \
&& for I in $(find /opt/Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip3 install -U \
-r $I || true; done

The answer is obvious Doctor Watson, right? These highly readable commands (pun intended) allow you to update your Cortex analyzers and responders to the latest stable versions, downloading new ones in the process, going over all the Python 2 and Python 3 dependencies to install the missing ones and upgrade the old ones to make sure they work correctly. These operations take quite a long time and cause some headaches in the process (Hello, I have Python 3.X and this dependency is no longer required, or Hi, I have an old version of Python 2 and it seems I need this other dependency).

And if you are lucky enough to get it running smoothly, you are still not done as you need to log in to the Cortex UI as an organisation administrator (unlike TheHive, Cortex supports multi-tenancy), click on the Refresh analyzers button under Organization > Analyzers then go to Organization > Responders and click on Refresh responders.

So while the answer to the opening question might be simple, updating analyzers and responders is far from being straightforward, to say the least, even if we forget the ugly fact that both are stored in a repository “conveniently” named Cortex-Analyzers*:

thehive@thehive-training:/opt/Cortex-Analyzers$ ls -d a* r*
analyzers  responders

Unnecessary Complexity Must Die

Your lovely, hard-working bees hate unnecessary complexity. Our project’s front page blatantly states our mission to bring Security Incident Response to the masses. And we have to stand by our words even if TheHive and Cortex are free, open source solutions and we do not gain anything from them save for the huge satisfaction of helping our fellow incident handlers level the fight against cybercriminals & all kinds of other animals of the APT (Advanced Persistent Troll Threat) bestiary.

There is only one possible solution: simplify the installation and update process of the current, official 115 analyzers and responders we have as of this writing, the future ones and any private or unofficial ones written in other programming languages such as those developed in Go by Rosetelecom-CERT.

Docker all the Things!

Starting from Cortex 3.0, the next major release of your favourite analysis and active response engine, all analyzers and responders will be dockerized. It will no longer be necessary to install them along with their various dependencies. They will be dowloaded from our cortexengine Docker organisation. Sysadmins might also configure automatic updates.

As a side advantage of using Docker, analyzers, and responders will also be isolated from each other which gives more flexibility and possibilities.

© Steve Simson. This artwork and the title of this blog post are inspired by The Revolution will not be televised, a song from the late and great Gil Scott-Heron.

For those users who have private, custom analyzers and responders that they don’t want or can’t share with the community, several options will be available:

  • Continue managing their analyzers and responders in the same way as currently supported by Cortex 2 (i.e. launch them as processes, with no isolation whatsoever).
  • Dockerize them and store them locally on their Cortex instance.
  • Dockerize them and publish them on a Docker registry, either the official one or a private registry.

A Docker image of Cortex 3 will still be provided. It will contain a Docker engine to launch dockerized analyzers and responders using DIND (Docker in Docker).

It won’t be necessary to modify the code of the current, official analyzers and responders. A drone job will monitor the analyzer and responder repository and automatically build docker images when it detects changes.

The Cortex Web interface will be slightly modified to accommodate the whole process and allow adding in-house/private Certificate Authorities to allow Cortex to smoothly perform updates in those corporate environments where TLS/SSL inspection is enabled.

Nice Movie Trailer. When is it Coming to a Theatre near me?

We are working hard to get Cortex 3 out of the oven in Q1 (of this year, yes). We will reach out to you, dear reader, in due time, to help us test it and refine it before putting it on the digital shelf for free, as usual. We will provide a smooth migration path in order to move safely your current analyzers and responders and their configuration to Cortex 3.

So to paraphrase the late and great Gil Scott-Heron:

The dockerization will not be televised

The dockerization will not be televised

The dockerization will be live.

Since you are here

The success of TheHive and Cortex continue to grow, far more than we initially foresaw. As far as we know, there are about a hundred organisations of different sizes and locations using or testing them. And as the number of users grows, so does the number of features, professional service and support requests.

We have tried addressing these requests through Creative Source, a nonprofit organisation (NPO). All but one company trusted us enough to make a donation and get tailored services for its needs in return. Most of the others either did not reply to our proposals or explained that their procurement process does not accommodate working with NPOs.

Some members of our core team are actively working on alternative options to ensure not only the viability of TheHive and Cortex as FOSS products on the long run but the ability to provide professional training, support, and services without freaking out highly bureaucratic, think-in-the-box-but-never-outside procurement departments.

Stay tuned 🐝


(*) When the idea behind Cortex was born into our hive mind, we did not initially think about active response capabilities. So we naturally called the repository which was supposed to contain analyzers Cortex-Analyzers . When, at a later stage, we added responders, we put them in the same repository for obvious laziness pretences  ¯\_(ツ)_/¯.

Correction: February 15, 2019
Typographical errors have been corrected. Some rewording has been made for the sake of clarity.

UnshortenLink, SSRF and Cortex-Analyzers 1.15.2

We urge our fellow fighters of cybercrime and funny and not-so-funny animals-in-disguise, to update their Cortex analyzers to the latest 1.15.2 version which plugs a serious vulnerability in the Unshortenlink analyzer and fixes several bugs.

Unshortenlink Vulnerability

On Jan 24, Alexandre Basquin discovered a SSRF vulnerability in the Unshortenlink 1.0 analyzer. By exploiting it, an attacker which has access to a Cortex instance with an analyze role can scan the ports of localhost and possibly of all the hosts reachable by Cortex. This could be automated through Cortex4py by making repetitive calls to the API and thus scanning multiple ports & hosts. In essence, the attacker can perform reconnaissance thanks to Unshortenlink and gain knowledge on which ports are open and which aren’t.

Version 1.1 of Unshortenlink, included in Cortex-Analyzers 1.15.2, plugs this vulnerability by disabling submissions containing IPs & ports. Only URLs are now accepted.

Bug Fixes

Additionally, Cortex-Analyzers 1.15.2 corrects the following bugs:

  • #416: the Mailer responder now supports UTF-8 encoding.
  • #410: remove wrapping of the results produced by crt.sh as a list which ends up breaking the output of the report template. Contributed by Thomas Kastner.
  • #409: enum is not required for Python 3.4+ to make the MISP Search analyzer work.
  • #408: FileInfo’s Manalyze plugin did not work after Manalyze renamed plugin_btcaddress to plugin_cryptoaddress.
  • #406: fix a broken link in the Cymon_Check_IP report, submitted by Manabu Niseki.
  • #313: fix wrong file handling in OTXQuery.

How to Update your Analyzers?

To update your analyzers to version 1.15.2, please read the Updating section of the Cortex installation guide.

Since the version numbers of few analyzers have changed, you’ll need to connect as an orgadmin to Cortex to refresh the analyzer list, disabling old ones and enabling the new versions.

TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu
  • click on Import templates button and select the downloaded package

Wait a Minute! Where’s the Blog Post about 1.15.1?

Good catch! There was no blog post about Cortex-Analyzers 1.15.1. Call us lazy but there was nothing Earth-shattering, pole-shifting in that release. Nonetheless, here are a list of fixes included in that release and from which you’ll inherit automatically if you update 1.15.0 to 1.15.2 directly:

  • #402: Malwares analyzer code relied on functionality that’s only available in Python 3.7+. It has been fixed to work with 3.4 and up thanks to the work of Arcuri Davide.
  • #404: fixes a bad folder renaming in the HIBP (Have I Been Pwned) analyzer.
  • #398: MISP Search analyzer wouldn’t run without the enum dependency. Contributed by Mars Huang. Later improved in 1.15.2 (see above).

No Luck Luke?

If something does not work as expected, keep calm and ask on our user forum, reach out to fellow users on Gitter, or send us an email at support@thehive-project.org.

TheHive 3.3-RC2, Hot out of the Oven

TheHive Project’s code Chefs, dressed in their outright haute cuisine outfit, including the traditional toque blanche, have been quite busy lately, working on dockerizing all the Cortex analyzers (more on this later in an upcoming post), and doing tedious work to prepare the replacement of Elasticsearch by a GraphDB which will help us finally release much-awaited features such as multi-tenancy, delayed for way too many months (yeah, yeah, don’t chastise them but feel free to help them). In the meantime, they found enough bandwidth to release a new major version of TheHive.

Version 3.3, currently a release candidate includes several bug fixes and many new features as outlined below. Please note that TheHive 3.3-RC2 is beta software. As all our other release candidates, you can grab it from the pre-release, beta repositories. As usual, we would truly appreciate your help making it a great stable release by testing it as thoroughly as possible and reporting back any bugs or issues you encounter so we can address them before the final release.

Check TheHive Installation guide for further details.

Wait! Where’s RC1?

TheHive 3.3-RC1 was very short-lived. Few hours after its release, and thanks to Chris (a.k.a. crackytsi on GitHub), Thomas Franco, our back-end mastermind, discovered an issue with the Debian 8 and Debian 9 packages.

New Features

  • #836: add a new exportCaseTags parameter to the MISP configuration section. If set to true, all the tags associated with a case will be exported along with it to MISP.
  • #861: add support for Java higher than 8, such as OpenJDK 11.
  • #271: bulk merge alerts into a case. Select multiple alerts at once and create a single case out of them or merge them into an existing case using its ID.
  • #824: add ability to sort alerts by reference, status, type, source…
  • #826: when previewing an alert, there are sometimes no overlap with an existing case. However, an analyst might already know, thanks to HI (Human Intelligence), that the alert should be merged into a specific case. This is now possible thanks to a new button.
  • #769: improve case template selection for case creation. If you have defined a large set of case templates, you will be able to sort/filter to find the case you want to use when creating a New Case.
New case template selector
  • #657: add observable tags auto-completion. Contributed by Tyler Chong (Thanks!).
Observable tag auto-completion

Fixed Bugs

  • #864: do not return a session cookie when making an API call.
  • #856: there was a bug where after a followed alert PATCH, if the alert has already been promoted to a case, the case is not updated. Now, if the alert has follow=true, if it gets updated, its status is set to Updated and the related case is updated too.
  • #845: assigned but unstarted tasks were not showing up in My Tasks.
  • #844: enable user account locking through the Delete API endpoint.

Stuck?

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

Cortex 2.1.3: Security and Beyond

TheHive Project’s code Chefs are happy to announce the immediate availability of Cortex 2.1.3, a hotfix for your favorite observable and response engine, fresh out of the oven!

We highly recommend that you upgrade your existing installation to this new version as soon as feasible as it plugs a significant security vulnerability, kindly reported by Po-Hsing Wu. The vulnerability is a privilege escalation one which allows an orgadmin to create a superadmin user. The culprit has been punished by having to chant Perl mantras while doing a handstand on burning coals.

security_holes
Source : XKCD

Additionally, Cortex 2.1.3 fixes the following bugs:

  • #157: list and disable invalid responders
  • #152: enforce PAP when launching an analyzer from the Cortex Web UI
  • #147: add dig to the Cortex docker image as the SinkDB analyzer needs it
  • #146: the Cortex job list must display the PAP value
  • #145: fix the broken Web UI’s search function for job history

Pardon my French but do you speak English?

Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

TheHive 3.2.0-RC1: The MISP Love Edition

Guess what? Our integration with MISP, the de facto standard for threat sharing, has just gotten better with our latest beta release: TheHive 3.2.0-RC1.

While you could synchronize TheHive with one or multiple MISP instances in earlier versions and select events using filters like their age, the number of attributes they contain or exclude those which are created by specific organisations or contain one or several black-listed tags, 3.2.0-RC1 adds the ability to whitelist tags, thus limiting the events that would show up in TheHive’s Alerts pane to only those which have been tagged with labels your SOC/CSIRT/CERT needs to act on. This can be very useful for example if your Cyber Threat Intelligence analysts pre-select or create events in MISP and tag for SOC consumption those that need to be acted on.

The Hive - Logo - Schéma - V1_Plan de travail 1.png
TheHive, Cortex, MISP: The Power DFIR & CTI Trio

To use this feature, use the whitelist.tags parameter in the MISP section of TheHive’s application.conf as described in the documentation.

This new version also adds the ability to create dashboards out of responder actions, log responder operations, and offers a confirmation dialog before running a responder to avoid noob over-clicks and errors made by seasoned incident handlers running low on caffeine.

TheHive 3.2.0-RC1 will also show you the description of an observable if any while hovering over one in the Observables tab. You can also see observable tags when previewing an alert in the Alerts pane.

Last but not least, some users reported severe problems when they enabled TLS/SSL directly on TheHive without resorting to a reverse proxy such as NGINX. Blame that on the crappy TLS support in Play framework ;-). So we highly recommend using a reverse proxy for that purpose, and delegate authentication to it if you are relying on X.509 authentication, as TheHive 3.2.0-RC1 allows you to. Please check the Single Sign-On on TheHive with X.509 Certificates guide for further information.

For additional details on this release, please check the full changelog.

Warning Capt’n Robinson!

The RC in 3.2.0-RC1 stands for Release Candidate. Please help us make a great stable release out of it by testing it as thoroughly as possible and reporting back any bugs or issues you encounter so we can address them before the final release. You’ll find this release candidate in the pre-release, beta repositories.

Please check TheHive Installation guide for further details.

You got a problem?

Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.