Lo and behold, we aren’t dead & TheHive Project ain’t toast! So, foremost, Happy New Year folks (we are still in January, right?)! We have some nice gifts coming up for you, gifts that have required very heavy-duty work. Of course, you might complain that we haven’t been responsive as of late but hey, there’s only so much we can do, right?
We’ll talk about those gifts in the upcoming weeks. In the meantime, there’s a new Cortex version in town and we urge you to upgrade to it, particularly if you consider deploying several Cortex nodes as a cluster. Indeed, Cortex 3.0.1 fixes a missing dependency that is required to set up such an architecture. Additionally (and this is the part where you should be paying attention), this version fixes the display of error messages pertaining to analyzer and responder operations, and also ensure that old responders and analyzers no longer show up once you clicked on that Refresh button.
Fixes and Enhancements
#244 Prevent the Play secret key from being displayed in the logs at startup. Nonetheless, you can still display it (for troubleshooting purposes or to make things easier for attackers that might have access to the logs and be interested in such a world-changing secret) by using the --show-secret option when starting Cortex
#243 fixes the display of error messages when analyzers & responders fail
One of the big improvements you’ll notice in Cortex 3 is the support for dockerized analyzers. And amongst some of their benefits, the installation process has been significantly simplified. So let’s assume you do not want to bang your head against Python, or other library dependencies. Then read one for a way to set up analyzers and run them quickly.
The following instructions have been tested on Ubuntu 18.04. If you already have a Cortex instance up and running, you can jump directly to the docker installation section below.
Install System Packages
Ensure your system contains the required packages:
Once Cortex is configured, restart the service with the following command, wait a few seconds and you should be able to connect to Cortex on https://<yourCortexHost>:9001 et voilà!
sudo service restart cortex
Important Note: The catalog analyzers.json contains information regarding versions of analyzers we consider stable and that are updated with bug fixes. This is typically synchronised with our master branch on Github. When you are using this catalog, you are de facto benefiting from the latest analyzer updates without needing to refresh anything in Cortex or setup again the configuration to get the latest version.
We also provide two additional catalogs:
analyzers-stable.json which strictly follows versions of analyzers if you do not want any uncontrolled updates. What does that mean in practice? You will have to click on the Refresh button in Cortex to update your analyzers, disable old ones and enable new versions. Moreover, you will also have to setup again their configuration. Typically, if you installed and setup Cortex with this catalog and the current version of FileInfo analyzers is 6.0, you won’t benefit from the next version, let’s say 6.1, unless you refresh Cortex.
analyzers-devel.jsonwhich contains information about new analyzers or version of analyzers that contains code that has been reviewed but not tested enough (or even not tested at all at times) to be deemed ready for production environments. This is typically synchronized with the develop branch of our Github repository.
Same goes for responders. All available catalogs for Cortex are published on bintray so you can choose the one that better fits your needs (or your risk/gambling profile :p).
For many months, we have been concentrating our efforts on TheHive 4, the next major version of your favourite Security Incident Response Platform, which we’ll finally provide RBAC (or multi-tenancy if you prefer), a feature that Cortex had for quite some time now.
As you well know, both TheHive and Cortex rely on Elasticsearch (ES) for storage. The choice of ES made sense in the beginning of the project but as we added additional features and had new ideas to give you the best experience possible, we faced several ES quirks and shortcomings that proved challenging if not outright blocking for making our roadmap a reality, including RBAC implementation in TheHive, a far more complex endeavour than RBAC in Cortex. Transitioning from ES to graph databases was necessary and since we want our existing users to have a smooth migration path, TheHive 4 (the first release candidate should come out of the oven by the end of the year) will support both ES and graph databases.
But while we were focusing on that, we completely lost sight of the end of life of ES 5.6 so we wrote an apology to you, our dear users, back in May.
Shortly after, we released TheHive 3.4.0-RC1, to add support for ES 6 (with all the breaking changes it has introduced). We also did the same for Cortex with the release of Cortex 3.0.0-RC3. We also took that opportunity to clear out some AngularJS technodebt we had.
We then asked you to take them for a spin and report back any bugs you find given that both versions had to support ES 5.6 and ES 6 to allow for proper migration.
After a few rounds of release candidates, we are pleased to announce the immediate availability of TheHive 3.4.0 and Cortex 3.0.0 as stable releases.
As we announced on May 14, 2019, we have been working very hard to add Elasticsearch 6 support to TheHive and Cortex as Elasticsearch 5.x went the way of the dodo when Elastic plugged life support off this venerable version. We also took this occasion to upgrade AngularJS and its sub projects to 1.7.8, the latest 1.x version as of this writing. Additionally, Grunt build dependencies have also been updated to their latest compatible versions.
It took us more time than initially foreseen but hey, we all love deadlines. We all love the whooshing noise they make as they go by.
TheHive 3.4.0-RC1 and Cortex 3.0.0-RC3 are now available on every Internet pipe near you and before you take them for a spin to help us identify any issues to make the stable releases rock-solid, let us walk you through some important information. Relax and grab a drink (and send good wine our way, we can always use some!).
In addition to ES5 and 6 support and the update of AngularJS, this version corrects a few bugs that were identified in the latest stable version (3.3.1) and adds a few features. The most important one in our opinion is the ability to import a file from a Cortex report. This requires Cortex 3.0.0-RC3. The full list of changes is available at the following location.
ES5 and ES6 support, AngularJS et cetera et cetera. Well you know the song right? Not quite as Cortex 3.0.0 significantly facilitates analyzer and responder installation and updates, thanks to Docker as we touched upon in a blog post earlier this year.
As detailed in the Cortex migration guide, which we recommend you read thoroughly, you can migrate from Cortex 2 and keep using analyzers and responders the same way (using processes), use the new Docker-based analyzers and responders or mix and match between running processes and docker containers (but then, you gotta pay extra attention to configure properly which analyzer/responder runs in which fashion).
Moreover, if you use the new dockerised analyzers and responders, you will be able to choose if you want to have them autoupdated (that’s the default behaviour) and if so, pick the bleeding edge, potentially buggy versions, the minor releases or, if you are risk-averse, stick with stable ones.
Cortex 3.0.0-RC3 also adds the ability to retrieve files resulting from analyzer jobs and last but not least, corrects an information disclosure bug that allowed non-admin users to retrieve the details of other users through the API. The vulnerability was reported by Adam Maris so kudos to him!
TheHive 3.4.0-RC1 and Cortex 3.0.0-RC3 use HTTP transport (9200/tcp by default) to connect to Elasticsearch instead of its native binary protocol (9300/tcp by default).
SSL/TLS, including when using a client certificate, can be configured to connect securely to ES. However this has not been tested yet.
Support of X-Pack and Search Guard is discontinued for anything but basic and SSL client authentication, which would still work.
Caution: Performance May Take a Hit!
The parent-child relationships we use behind the scene in Elasticsearch could make queries significantly slower with ES 6 and in our limited testing, we had the impression that performance took a hit. So please be cautious there and we’d be grateful if you could report any sluggishness you notice during your tests of the new versions with ES6.
Spring is here and your favorite bees are busy buzzing flowers to prepare you the most palatable honey ever. In the meantime, we are pleased to announce the immediate availability of Cortex-Analyzers 1.16.0, which adds a new responder and three new analyzers to complete an already hefty collection, bringing the total to 117 analyzers and 3 responders!
One responder has been added:
FalconCustomIOC, contributed by Michael (#421). We don’t know the last name of Michael. That could be Jordan. Who knows?
We could not duly test some of these additions due to lack of access to the associated services or to our legendary laziness. So we would really appreciate it if you could test them and let us know whether they work or not.
The FalconCustomIOC responder let you submit observables from alerts or cases to Crowdstrike Falcon Custom IOC service.
Crowdstrike Falcon is a paid service. An account and an API key are required to configure and run this responder.
AbuseIPDB analyzer let you determine wether an IP has been reported as malicious or not to the AbuseIPDB web service.
An account and an API key is needed to configure and use this analyzer.
TheHive displays the analyzer results as follows:
The BackscatterIO analyzer lets you query the Backscatter.io service for IPs, networks or autonomous systems (AS). It comes in two flavors:
BackscatterIO_GetObservations: determine whether an observables has a known scanning activity
BackscatterIO_Enrichment: enrich your observables with additional information
TheHive displays this analyzer results as follow:
SoltraEdge analyzer lets you query any observable against theSoltra Edge platform.
To configure and use this analyzer, an account, a token key and the base URL of a SoltraEdge server are needed.
TheHive displays this analyzers result as follow:
Get It While Supply Lasts!
Each analyzer and responder comes with its own, pip compatible requirements.txt file. To update your Cortex analyzers to 1.16.0, run the following commands:
cd path/to/Cortex-Analyzers git pull for I in analyzers/*/requirements.txt; do sudo -H pip2 install -U -r $I || true; done && \ for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done
On February 10, 2019, we released TheHive 3.3-RC2. It contained new features such as bulk alert merging, alert sorting, observable tag autocompletion, exporting case tags to MISP & more. Since then your favourite French code Chefs have been beesy refining TheHive 3.3 through new release candidates while getting Cortex 3 ready for prime time.
Over the weekend, Nabil decided he was not working enough already during the week. So he drained his batteries to the very last drop to release TheHive 3.3-RC5 before he crashed headfirst into his bed for a long, reparative sleep. Cumulatively since RC2, we added several features and squashed 10 bugs as described below.
Note that release candidates are beta software. You can get TheHive 3.3-RC5 from the pre-release, beta repositories. As usual, we encourage you to test it and report any bugs or issues you spot so we can address them before the final release.
#888: add a new UI configuration admin section. One of the first use cases of this section consist in disabling creating empty cases (i.e. cases not associated with a template). It will be gradually improved with new use cases so speak your mind!
#893: disable the case template selection when trying to merge multiple alerts for which no case template exists.
Do you know what the following set of commands achieve?
$ cd /opt/Cortex-Analyzers
$ sudo git pull
$ for I in $(find /opt/Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip2 install -U -r $I; done \
&& for I in $(find /opt/Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip3 install -U \
-r $I || true; done
The answer is obvious Doctor Watson, right? These highly readable commands (pun intended) allow you to update your Cortexanalyzers and responders to the latest stable versions, downloading new ones in the process, going over all the Python 2 and Python 3 dependencies to install the missing ones and upgrade the old ones to make sure they work correctly. These operations take quite a long time and cause some headaches in the process (Hello, I have Python 3.X and this dependency is no longer required, or Hi, I have an old version of Python 2 and it seems I need this other dependency).
And if you are lucky enough to get it running smoothly, you are still not done as you need to log in to the Cortex UI as an organisation administrator (unlike TheHive, Cortex supports multi-tenancy), click on the Refresh analyzers button under Organization > Analyzers then go to Organization > Responders and click on Refresh responders.
So while the answer to the opening question might be simple, updating analyzers and responders is far from being straightforward, to say the least, even if we forget the ugly fact that both are stored in a repository “conveniently” named Cortex-Analyzers*:
thehive@thehive-training:/opt/Cortex-Analyzers$ ls -d a* r*
Unnecessary Complexity Must Die
Your lovely, hard-working bees hate unnecessary complexity. Our project’s front page blatantly states our mission to bring Security Incident Response to the masses. And we have to stand by our words even if TheHive and Cortex are free, open source solutions and we do not gain anything from them save for the huge satisfaction of helping our fellow incident handlers level the fight against cybercriminals & all kinds of other animals of the APT (Advanced Persistent Troll Threat) bestiary.
There is only one possible solution: simplify the installation and update process of the current, official 115 analyzers and responders we have as of this writing, the future ones and any private or unofficial ones written in other programming languages such as those developed in Go by Rosetelecom-CERT.
Docker all the Things!
Starting from Cortex 3.0, the next major release of your favourite analysis and active response engine, all analyzers and responders will be dockerized. It will no longer be necessary to install them along with their various dependencies. They will be dowloaded from our cortexengine Docker organisation. Sysadmins might also configure automatic updates.
As a side advantage of using Docker, analyzers, and responders will also be isolated from each other which gives more flexibility and possibilities.
For those users who have private, custom analyzers and responders that they don’t want or can’t share with the community, several options will be available:
Continue managing their analyzers and responders in the same way as currently supported by Cortex 2 (i.e. launch them as processes, with no isolation whatsoever).
Dockerize them and store them locally on their Cortex instance.
Dockerize them and publish them on a Docker registry, either the official one or a private registry.
A Docker image of Cortex 3 will still be provided. It will contain a Docker engine to launch dockerized analyzers and responders using DIND (Docker in Docker).
It won’t be necessary to modify the code of the current, official analyzers and responders. A drone job will monitor the analyzer and responder repository and automatically build docker images when it detects changes.
The Cortex Web interface will be slightly modified to accommodate the whole process and allow adding in-house/private Certificate Authorities to allow Cortex to smoothly perform updates in those corporate environments where TLS/SSL inspection is enabled.
Nice Movie Trailer. When is it Coming to a Theatre near me?
We are working hard to get Cortex 3 out of the oven in Q1 (of this year, yes). We will reach out to you, dear reader, in due time, to help us test it and refine it before putting it on the digital shelf for free, as usual. We will provide a smooth migration path in order to move safely your current analyzers and responders and their configuration to Cortex 3.
The success of TheHive and Cortex continue to grow, far more than we initially foresaw. As far as we know, there are about a hundred organisations of different sizes and locations using or testing them. And as the number of users grows, so does the number of features, professional service and support requests.
We have tried addressing these requests through Creative Source, a nonprofit organisation (NPO). All but one company trusted us enough to make a donation and get tailored services for its needs in return. Most of the others either did not reply to our proposals or explained that their procurement process does not accommodate working with NPOs.
Some members of our core team are actively working on alternative options to ensure not only the viability of TheHive and Cortex as FOSS products on the long run but the ability to provide professional training, support, and services without freaking out highly bureaucratic, think-in-the-box-but-never-outside procurement departments.
Stay tuned 🐝
— (*) When the idea behind Cortex was born into our hive mind, we did not initially think about active response capabilities. So we naturally called the repository which was supposed to contain analyzers Cortex-Analyzers . When, at a later stage, we added responders, we put them in the same repository for obvious laziness pretences ¯\_(ツ)_/¯.
Correction: February 15, 2019 Typographical errors have been corrected. Some rewording has been made for the sake of clarity.