Searching for an Elastic? Here, Take 6!

As we announced on May 14, 2019, we have been working very hard to add Elasticsearch 6 support to TheHive and Cortex as Elasticsearch 5.x went the way of the dodo when Elastic plugged life support off this venerable version. We also took this occasion to upgrade AngularJS and its sub projects to 1.7.8, the latest 1.x version as of this writing. Additionally, Grunt build dependencies have also been updated to their latest compatible versions.

It took us more time than initially foreseen but hey, we all love deadlines. We all love the whooshing noise they make as they go by.

TheHive 3.4.0-RC1 and Cortex 3.0.0-RC3 are now available on every Internet pipe near you and before you take them for a spin to help us identify any issues to make the stable releases rock-solid, let us walk you through some important information. Relax and grab a drink (and send good wine our way, we can always use some!).

Source: https://dilbert.com/strip/1995-11-10

TheHive 3.4.0-RC1

In addition to ES5 and 6 support and the update of AngularJS, this version corrects a few bugs that were identified in the latest stable version (3.3.1) and adds a few features. The most important one in our opinion is the ability to import a file from a Cortex report. This requires Cortex 3.0.0-RC3. The full list of changes is available at the following location.

Prior to migrating to 3.4.0-RC1, please read the migration guide.

Cortex 3.0.0-RC3

ES5 and ES6 support, AngularJS et cetera et cetera. Well you know the song right? Not quite as Cortex 3.0.0 significantly facilitates analyzer and responder installation and updates, thanks to Docker as we touched upon in a blog post earlier this year.

As detailed in the Cortex migration guide, which we recommend you read thoroughly, you can migrate from Cortex 2 and keep using analyzers and responders the same way (using processes), use the new Docker-based analyzers and responders or mix and match between running processes and docker containers (but then, you gotta pay extra attention to configure properly which analyzer/responder runs in which fashion).

Moreover, if you use the new dockerised analyzers and responders, you will be able to choose if you want to have them autoupdated (that’s the default behaviour) and if so, pick the bleeding edge, potentially buggy versions, the minor releases or, if you are risk-averse, stick with stable ones.

Cortex 3.0.0-RC3 also adds the ability to retrieve files resulting from analyzer jobs and last but not least, corrects an information disclosure bug that allowed non-admin users to retrieve the details of other users through the API. The vulnerability was reported by Adam Maris so kudos to him!

Warning: Regressions Ahead!

As outlined in our previous post about these new versions:

  • TheHive 3.4.0-RC1 and Cortex 3.0.0-RC3 use HTTP transport (9200/tcp by default) to connect to Elasticsearch instead of its native binary protocol (9300/tcp by default).
  • SSL/TLS, including when using a client certificate, can be configured to connect securely to ES. However this has not been tested yet.
  • Support of X-Pack and Search Guard is discontinued for anything but basic and SSL client authentication, which would still work.

Caution: Performance May Take a Hit!

The parent-child relationships we use behind the scene in Elasticsearch could make queries significantly slower with ES 6 and in our limited testing, we had the impression that performance took a hit. So please be cautious there and we’d be grateful if you could report any sluggishness you notice during your tests of the new versions with ES6.

An Apology

Dear Users,

We owe you an apology. We thought we would never need to support Elasticsearch 7 or even 6. We thought we could stick with the latest version of Elasticsearch 5 as the underlying storage and indexing engine for TheHive and Cortex until we would be able to complete the transition to a graph database. Moving to such a database is a necessity for your favourite open source, free Security Incident Response Platform and its analysis and orchestration companion, a necessity that has grown out of our frustration with Elasticsearch and its limitations, with the breaking changes that ES 6 introduced which forbid a smooth transition and puts a significant toll on an open source initiative such as ours.

We initially thought we could complete the transition by October of last year and finally offer you long-desired features such as RBAC and multi-tenancy as well as establish a solid ground to implement some exciting ideas that would help you lower the barrier to entry for junior analysts, save more time and concentrate on your work instead of having to master copy/paste between various interfaces or moving from one tool to the other.

Sadly, things did not play out the way we wanted. As TheHive and Cortex were adopted by more and more organisations, feature requests kept piling up and being generous bees, we have always strived to keep our users happy within the confines of our limited resources. Certainly, our user community helped us significantly by contributing a huge number of analyzers to Cortex in no time, making the total amount fly past the 100 landmark. However, we had to rely mostly on ourselves for heavy-duty backend work while steadily releasing new versions to satisfy the appetite for capabilities that sounded reasonable and feasible within a realistic, acceptable timeframe. Multi-tenancy and RBAC also proved more complex than initially foreseen and since we hate a half-baked recipe (blame it on our French culture and our love for delicious food), we did not want to rush things out and add flimsy ‘patch’ code.

Source : https://kininaru-korean.net/archives/10305

So we focused on supporting graph databases and working on multi-tenancy and RBAC. You certainly noticed our silence these past weeks. And we completely lost sight of the end of life of ES 5.6 until we realised recently that it was no longer supported by Elastic, not even in critical bug fix mode. When ES 7 was released on April 10, the death sentence of ES 5.6 was pronounced and its coffin permanently nailed.

We know this is a lot to stomach. Welcome to the Upside Down! But remember: keep calm. Help is already on the way and hopefully this time around the cops will arrive before the movie is over. We are shifting our priorities to release new major versions of TheHive and Cortex in order to use a supported version of ES. This work should take a few weeks at least. In the meantime, if you are using TheHive and Cortex with their own, standalone ES instance and you have implemented sane network security measures to shield ES against unwanted remote access, you should be fine.

We also took the opportunity to look at what other external code we rely on and that would need to be updated as well, to avoid falling in the EOL trap again. Glad we looked! The current versions of TheHive and Cortex both use AngularJS 1.5 (here, take a stone and throw it the Hulk’s way on Nabil’s forehead). We are going to update our frontends to use AngularJS 1.7.

We will come up imminently with a concrete action plan to address our embarrassing miscalculation. Meanwhile, please accept our sincere apologies and rest assured that we won’t let you down.

ごめんなさい 🙏🏼