Soon after we released Cortex-Analyzers 2.4.0, Jérôme noticed that something was definitely wrong. And that something was plural.
As he set to retest a few things here and there, he realised that many docker images, for the latest and greatest analyzers and responders, were not automatically built. The code factory wasn’t working 😰
So he started digging. And the more he dug, the more bugs he discovered. Our user community also reported a few issues. He thought it was about time he opens that bottle of Aloxe-Corton, put a Makaya McCraven album to play on his turntable, and rolls his sleeves to address all these problems headfirst 🍷
After a few hours of intense work, he managed to fix the docker build process and release Cortex-Analyzers 2.4.1, a hotfix that corrects the following issues:
[#545] Message extraction using FileInfo doesn’t always work
[#610] The VirusTotal analyzer contains a typo which prevents it from running
[#614] Many analyzers fail to run due to incorrect permissions
[#619] Abuse Finder not working with docker after force usage of python3
[#620] Missing library prevented the build of the docker image corresponding to the new MalwareClustering analyzer
Finally, he took the opportunity to rename Palo Alto AUTOFOCUS analyzers to Autofocus, for consistency purposes.
Please refer to our previous blog post, pertaining to Cortex-Analyzers 2.4.0, for update instructions.
Thank you in advance for your understanding and happy cyberfighting! 💪🏼
Guess what? TheHive Project is still alive and well, as Saâd already mentioned in a previous blog post.
We’ve been certainly very busy lately, preparing the upcoming release of TheHive 4 and doing many other things beside working on our FOSS project. As a result, it took us a rather long time to merge several community contributions and reduce the sizeable pile of pull requests.
We would like to thank our contributors for their patience and we hope the cyberdefenders out there will enjoy the brand new Cortex-Analyzers 2.4.0 release, with many new analyzers, responders and some bug fixes & improvements, bringing the total to a whooping 138 analyzers (counting all flavors) and 10 responders!
Additionally, with this release, all analyzers are now using Python 3. No more Python 2 technodebt!
The EmailRep analyzer checks the reputation of an email address against the emailrep.io database.
This analyzer accesses IP-centric features provided by ipinfo.io. While the EmailRep API can be used without a token for limited usage, the ipinfo.io analyzer requires the configuration of an API token before use.
This analyzer lets you query the free Maltiverse Threat Intelligence platform for enrichment information about a particular hash, domain, ip or url.
TheHive displays the analyzer results as follows:
Andrea Garavaglia contributed this one a long time ago and we finally merged it into the Cortex-Analyzers repository. Andrea gave a talk about the background of this analyzer at the fourth MISP summit. You can watch it here.
In order to use the analyzer, you need to point it to a Neo4j server (you need to supply the host, port, login & password).
This analyzer lets you leverage PaloAlto Autofocus services. Provided you are an Autofocus customer and you have access to their API, you need to configure the analyzer with your username and a token key.
The analyzer comes with 3 flavors:
AUTOFOCUS_GetSampleAnalysis lets you request a full report for a given hash.
AUTOFOCUS_SearchIOC lets you research for samples linked to specific IoCs with datatypes like domain, fqdn, user-agent, imphash, ip, mutex, tag and url. Please note that mutex and tag are not default datatypes in TheHive. You need to create them in TheHive before you can leverage them.
AUTOFOCUS_SearchJSON lets you research for samples based on a complex JSON query.
Important: TheHive has no templates corresponding to this analyzer have been published yet. They will be provided in the near future.
This analyzer queries Team Cymru’s Malware Hash Registry for known malware hashes (MD5 or SHA-1). If it is malware and known by the service, it returns the last time it has been seen along with an approximate anti-virus detection percentage.
Overview of the New Responders
This responder allows the integration between TheHive/Cortex and KnowBe4’s User Events API. If a mail observable is tagged with a specified tag, corresponding to the responder’s configuration (e.g. phished), then the associated user will have a custom event added to their profile in KnowBe4.
A valid account on KnowBe4 and an API key are required to run this responder.
This responder sends observables you select to a Palo Alto Minemeld instance.
This responder performs actions on Wazuh, the open source security monitoring platform. It currently supports ad-hoc firewall blocking of ip observables.
New PassiveTotal flavors
Thanks to Brandon Dixon, the PassiveTotal analyzer gains 3 new flavors, bringing the total to 11:
PassiveTotal_Trackerslet you make tracker lookups on observables of type domain, fqdn and ip.
PassiveTotal_Host_Pairs let you make host pair lookups on observables of type domain, fqdn and ip.
PassiveTotal_Componentslets you make components lookup on observables of type domain, fqdn and ip.
They come with their own report templates.
The analyzer has been updated to support GreyNoise API v2, thanks to the contribution of Whitney Champion (#562).
New Data Types Supported by Some Analyzers
VirusTotal_GetReporthas been updated to allow requests for observables of type fqdn.
Threatcrowd has been updated to allow requests for observables of type domain.
Shodan has been updated to allow requests for observables of type fqdn.
[#602] The MISP analyzer was bumped to version 2.1 and is ready to use PyMISP 2.4.120.
Get It While Supply Lasts!
If you are using the dockerized analyzers & responders, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button.
If you are still using the old-style way of installing analyzers and responders, run the following commands:
cd path/to/Cortex-Analyzers git pull for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done
Once done, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button. Refer to the online Cortex documentation for further details.
Update TheHive Report Templates
If you are using TheHive, you must import the new report templates in your instance as follows:
For many months, we have been concentrating our efforts on TheHive 4, the next major version of your favourite Security Incident Response Platform, which we’ll finally provide RBAC (or multi-tenancy if you prefer), a feature that Cortex had for quite some time now.
As you well know, both TheHive and Cortex rely on Elasticsearch (ES) for storage. The choice of ES made sense in the beginning of the project but as we added additional features and had new ideas to give you the best experience possible, we faced several ES quirks and shortcomings that proved challenging if not outright blocking for making our roadmap a reality, including RBAC implementation in TheHive, a far more complex endeavour than RBAC in Cortex. Transitioning from ES to graph databases was necessary and since we want our existing users to have a smooth migration path, TheHive 4 (the first release candidate should come out of the oven by the end of the year) will support both ES and graph databases.
But while we were focusing on that, we completely lost sight of the end of life of ES 5.6 so we wrote an apology to you, our dear users, back in May.
Shortly after, we released TheHive 3.4.0-RC1, to add support for ES 6 (with all the breaking changes it has introduced). We also did the same for Cortex with the release of Cortex 3.0.0-RC3. We also took that opportunity to clear out some AngularJS technodebt we had.
We then asked you to take them for a spin and report back any bugs you find given that both versions had to support ES 5.6 and ES 6 to allow for proper migration.
After a few rounds of release candidates, we are pleased to announce the immediate availability of TheHive 3.4.0 and Cortex 3.0.0 as stable releases.
As we announced on May 14, 2019, we have been working very hard to add Elasticsearch 6 support to TheHive and Cortex as Elasticsearch 5.x went the way of the dodo when Elastic plugged life support off this venerable version. We also took this occasion to upgrade AngularJS and its sub projects to 1.7.8, the latest 1.x version as of this writing. Additionally, Grunt build dependencies have also been updated to their latest compatible versions.
It took us more time than initially foreseen but hey, we all love deadlines. We all love the whooshing noise they make as they go by.
TheHive 3.4.0-RC1 and Cortex 3.0.0-RC3 are now available on every Internet pipe near you and before you take them for a spin to help us identify any issues to make the stable releases rock-solid, let us walk you through some important information. Relax and grab a drink (and send good wine our way, we can always use some!).
In addition to ES5 and 6 support and the update of AngularJS, this version corrects a few bugs that were identified in the latest stable version (3.3.1) and adds a few features. The most important one in our opinion is the ability to import a file from a Cortex report. This requires Cortex 3.0.0-RC3. The full list of changes is available at the following location.
ES5 and ES6 support, AngularJS et cetera et cetera. Well you know the song right? Not quite as Cortex 3.0.0 significantly facilitates analyzer and responder installation and updates, thanks to Docker as we touched upon in a blog post earlier this year.
As detailed in the Cortex migration guide, which we recommend you read thoroughly, you can migrate from Cortex 2 and keep using analyzers and responders the same way (using processes), use the new Docker-based analyzers and responders or mix and match between running processes and docker containers (but then, you gotta pay extra attention to configure properly which analyzer/responder runs in which fashion).
Moreover, if you use the new dockerised analyzers and responders, you will be able to choose if you want to have them autoupdated (that’s the default behaviour) and if so, pick the bleeding edge, potentially buggy versions, the minor releases or, if you are risk-averse, stick with stable ones.
Cortex 3.0.0-RC3 also adds the ability to retrieve files resulting from analyzer jobs and last but not least, corrects an information disclosure bug that allowed non-admin users to retrieve the details of other users through the API. The vulnerability was reported by Adam Maris so kudos to him!
TheHive 3.4.0-RC1 and Cortex 3.0.0-RC3 use HTTP transport (9200/tcp by default) to connect to Elasticsearch instead of its native binary protocol (9300/tcp by default).
SSL/TLS, including when using a client certificate, can be configured to connect securely to ES. However this has not been tested yet.
Support of X-Pack and Search Guard is discontinued for anything but basic and SSL client authentication, which would still work.
Caution: Performance May Take a Hit!
The parent-child relationships we use behind the scene in Elasticsearch could make queries significantly slower with ES 6 and in our limited testing, we had the impression that performance took a hit. So please be cautious there and we’d be grateful if you could report any sluggishness you notice during your tests of the new versions with ES6.
We owe you an apology. We thought we would never need to support Elasticsearch 7 or even 6. We thought we could stick with the latest version of Elasticsearch 5 as the underlying storage and indexing engine for TheHive and Cortex until we would be able to complete the transition to a graph database. Moving to such a database is a necessity for your favourite open source, free Security Incident Response Platform and its analysis and orchestration companion, a necessity that has grown out of our frustration with Elasticsearch and its limitations, with the breaking changes that ES 6 introduced which forbid a smooth transition and puts a significant toll on an open source initiative such as ours.
We initially thought we could complete the transition by October of last year and finally offer you long-desired features such as RBAC and multi-tenancy as well as establish a solid ground to implement some exciting ideas that would help you lower the barrier to entry for junior analysts, save more time and concentrate on your work instead of having to master copy/paste between various interfaces or moving from one tool to the other.
Sadly, things did not play out the way we wanted. As TheHive and Cortex were adopted by more and more organisations, feature requests kept piling up and being generous bees, we have always strived to keep our users happy within the confines of our limited resources. Certainly, our user community helped us significantly by contributing a huge number of analyzers to Cortex in no time, making the total amount fly past the 100 landmark. However, we had to rely mostly on ourselves for heavy-duty backend work while steadily releasing new versions to satisfy the appetite for capabilities that sounded reasonable and feasible within a realistic, acceptable timeframe. Multi-tenancy and RBAC also proved more complex than initially foreseen and since we hate a half-baked recipe (blame it on our French culture and our love for delicious food), we did not want to rush things out and add flimsy ‘patch’ code.
So we focused on supporting graph databases and working on multi-tenancy and RBAC. You certainly noticed our silence these past weeks. And we completely lost sight of the end of life of ES 5.6 until we realised recently that it was no longer supported by Elastic, not even in critical bug fix mode. When ES 7 was released on April 10, the death sentence of ES 5.6 was pronounced and its coffin permanently nailed.
We know this is a lot to stomach. Welcome to the Upside Down! But remember: keep calm. Help is already on the way and hopefully this time around the cops will arrive before the movie is over. We are shifting our priorities to release new major versions of TheHive and Cortex in order to use a supported version of ES. This work should take a few weeks at least. In the meantime, if you are using TheHive and Cortex with their own, standalone ES instance and you have implemented sane network security measures to shield ES against unwanted remote access, you should be fine.
We also took the opportunity to look at what other external code we rely on and that would need to be updated as well, to avoid falling in the EOL trap again. Glad we looked! The current versions of TheHive and Cortex both use AngularJS 1.5 (here, take a stone and throw it the Hulk’s way on Nabil’s forehead). We are going to update our frontends to use AngularJS 1.7.
We will come up imminently with a concrete action plan to address our embarrassing miscalculation. Meanwhile, please accept our sincere apologies and rest assured that we won’t let you down.
On February 10, 2019, we released TheHive 3.3-RC2. It contained new features such as bulk alert merging, alert sorting, observable tag autocompletion, exporting case tags to MISP & more. Since then your favourite French code Chefs have been beesy refining TheHive 3.3 through new release candidates while getting Cortex 3 ready for prime time.
Over the weekend, Nabil decided he was not working enough already during the week. So he drained his batteries to the very last drop to release TheHive 3.3-RC5 before he crashed headfirst into his bed for a long, reparative sleep. Cumulatively since RC2, we added several features and squashed 10 bugs as described below.
Note that release candidates are beta software. You can get TheHive 3.3-RC5 from the pre-release, beta repositories. As usual, we encourage you to test it and report any bugs or issues you spot so we can address them before the final release.
#888: add a new UI configuration admin section. One of the first use cases of this section consist in disabling creating empty cases (i.e. cases not associated with a template). It will be gradually improved with new use cases so speak your mind!
#893: disable the case template selection when trying to merge multiple alerts for which no case template exists.
TheHive Project’s code Chefs, dressed in their outright haute cuisine outfit, including the traditional toque blanche, have been quite busy lately, working on dockerizing all the Cortex analyzers (more on this later in an upcoming post), and doing tedious work to prepare the replacement of Elasticsearch by a GraphDB which will help us finally release much-awaited features such as multi-tenancy, delayed for way too many months (yeah, yeah, don’t chastise them but feel free to help them). In the meantime, they found enough bandwidth to release a new major version of TheHive.
Version 3.3, currently a release candidate includes several bug fixes and many new features as outlined below. Please note that TheHive 3.3-RC2 is beta software. As all our other release candidates, you can grab it from the pre-release, beta repositories. As usual, we would truly appreciate your help making it a great stable release by testing it as thoroughly as possible and reporting back any bugs or issues you encounter so we can address them before the final release.
TheHive 3.3-RC1 was very short-lived. Few hours after its release, and thanks to Chris (a.k.a. crackytsi on GitHub), Thomas Franco, our back-end mastermind, discovered an issue with the Debian 8 and Debian 9 packages.
#836: add a new exportCaseTags parameter to the MISP configuration section. If set to true, all the tags associated with a case will be exported along with it to MISP.
#861: add support for Java higher than 8, such as OpenJDK 11.
#271: bulk merge alerts into a case. Select multiple alerts at once and create a single case out of them or merge them into an existing case using its ID.
#824: add ability to sort alerts by reference, status, type, source…
#826: when previewing an alert, there are sometimes no overlap with an existing case. However, an analyst might already know, thanks to HI (Human Intelligence), that the alert should be merged into a specific case. This is now possible thanks to a new button.
#769: improve case template selection for case creation. If you have defined a large set of case templates, you will be able to sort/filter to find the case you want to use when creating a New Case.
#657: add observable tags auto-completion. Contributed by Tyler Chong (Thanks!).
#864: do not return a session cookie when making an API call.
#856: there was a bug where after a followed alert PATCH, if the alert has already been promoted to a case, the case is not updated. Now, if the alert has follow=true, if it gets updated, its status is set to Updated and the related case is updated too.
#845: assigned but unstarted tasks were not showing up in My Tasks.
#844: enable user account locking through the Delete API endpoint.