Cortex, a free, open source software allows security analysts and threat hunters to analyze and enrich observables (IP addresses, hashes, domains, …) collected in the course of an investigation or received from third parties, for example through MISP, the de facto standard for threat sharing.
On March 29, 2018, we released Cortex 2, a major improvement over the previous version which brought, among other cool features, authentication, caching, multi-tenancy (RBAC) and rate limiting. Instead of deploying several Cortex 1 instances behind reverse proxies which would implement authentification, administrators can deploy a single Cortex 2, create multiple organizations and serve the needs of various information security populations while enjoying extra features.
On May 31, 2018, we published a brand new API guide so that developers can take advantage of the powerful REST API of the product. Sadly, Cortex4py, the FOSS Python library we provide to interact with the API was not compatible with Cortex 2. Until today.
Thanks to the hard work of our dear Nabil Adouani, we are happy to announce the immediate availability of Cortex4py 2.0.0, a complete rewrite of the library in Python 3. Cortex4py 2.0.0 is fully compatible with Cortex 2. However, it doesn’t work with Cortex 1.
While TheHive, the highly popular free and open source Security Incident Response Platform (SIRP) we develop has native support for many Cortex 2 instances, Python developers can leverage Cortex4py to interact with Cortex 2, manage organizations, users, analyzer configurations and analyze observables at scale from alternative SIRPs, SIEMs or custom scripts thanks to the 83 analyzers Cortex 2 has as of June 18, 2018.
Cortex 2: there is more than one way to interact with it
Use It
To install Cortex4py, use PIP3:
$ sudo -H pip3 install cortex4py
If you are using Python on a Windows operating system, please forgo the sudo command.
Usage
Cortex4py 2 comes with a usage guide which includes many examples. For example, if you want to fetch the last 10 successful jobs that have been executed against domain names and display the result summaries of those 10 jobs you could write something like:
Sample Python3 code to retrieve Cortex analyzer results
Migrating from Cortex4py 1
If you have already written scripts using Cortex4py 1.x (for Cortex 1), we tried to keep the already available methods. However, we recommend you adapt your code to leverage the new Cortex4py 2 classes and methods as soon as feasible. Moreover, the existing scripts must be updated to support authentication if you intend to use them with Cortex 2. Please read the Cortex4py 2 usage guide for more information.
Support
Cortex 2.0.0 is brand new software. As such, it might contain bugs and limitations. If you find any or encounter problems, please ask on our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.
Rejoice DFIR fighters! Your favorite Security Incident Response Platform has just gotten better. TheHive 3.0.10 (codename Cerana 0.10) is just out of the oven. Thomas and Nabil, as the true code Chefs they are, have fixed a number of issues and added a few enhancements in this release. Read on, update and defend your constituency!
Fixes
#446: keep observable reports when performing a case merge operation
#592: fix the error message when an observable is created with invalid data
#579: rotate application.log to keep it from becoming huge
#564: when submitting a job to Cortex, send the case ID as well
#563: poll the status of connectors (MISP, Cortex) every minute and update the UX display accordingly
#548: sort related cases by the amount of shared observables
#546: display start and end times for closed tasks
Support
Something does not work as expected? You have troubles installing or upgrading? You have spotted new bugs? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.
We are proud to announce the immediate availability of Cerana 0.9 (TheHive 3.0.9) and Cortex 2.0.4. These hotfix releases address a number of issues and we encourage you to update your current installation at your earliest opportunity. For your comfort and sanity. Seriously.
We also took this opportunity to update Cortex analyzers to fix issues with CIRCL Passive SSL, Hybrid Analysis, and the Joe Sandbox URL Analysis template. Moreover, we have updated the cortexutils library to set the taxonomy level to info if it is invalid. To upgrade cortexutils to 1.2.4:
Note: the Bluecoat analyzer was removed since it does not comply with the updated Terms of Service of Symantec Web Pulse SiteReview. Symantec does no longer permit programmatic querying of the service.
Fixes in Cerana 0.9
#527: display long reports when the analyst clicks on the corresponding short reports. Meh!
#541: make the drop-down menu for case templates scroll when there is a truckload of them.
Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.
Correction: April 14, 2018 An earlier version of this post did not mention that the Bluecoat analyzer was removed in the latest Cortex Analyzers repository release.
After announcing Cortex 2.0.0 and TheHive 3.0.7, the first version of your favorite SIRP that is (supposedly) compatible with the brand-new version of Cortex, last week, we thought it was time to relax and enjoy the upcoming, long Easter weekend, the sunny sky of Paris (if you can pierce the veil of the Forever Grey Cloud™ that is hanging over the city of lights), and great jazz music. Heck, I even tweeted about it … only to be proven wrong by Life (and Murphy).
We literally field tested Cortex 2 for 3 weeks, we squashed bugs here and there, until almost the very last minute before the release. And yet, our QA needs to be improved by leaps and bounds as we had to release Cortex 2.0.1 one day after unveiling 2.0.0 to correct some additional bugs. And then some members of the core team and of our growing user community took it for a spin. And all hell broke lose. Well, almost 🙂
Session collisions (when TheHive and Cortex 2 are used on the same machine), analyzer malfunctions, connectivity problems … issues that were not identified during the testing phase, even in a production environment, where everything worked as expected. And we call this ‘Computer Science’. Right, right…
So we worked hard, took out our Code Hammer (it’s like Thor’s but cyber) and blasted away all the bugs that we found out or that were reported to us (arigato gozaimasu!) and we are happy to announce the immediate availability of Cortex 2.0.2, TheHive 3.0.8, Cortexutils 1.2.3 and Cortex-Analyzers 1.9.2.
TL;DR Install or upgrade Cortex 2.0.2, update Cortexutils, git pull the Cortex-analyzers repo to get the latest version of the repository, upgrade to TheHive 3.0.8, follow the Quick Start Guide and have a drink.
If you have time (which is admittedly quite scarce nowadays), please read on the changelogs:
As stated in the previous post, we will release a new version of Cortex4py in order to make it compatible with Cortex 2, continue the work we started with our MISP Project friends to support MISP attribute enrichment through Cortex 2 (MISP currently only supports enrichment using Cortex 1), and perform a long-overdue overhaul of our documentation. We will also release a brand new version of TheHive4py.
Last but not least, we’ll take a hard look at ourselves and our QA. You expect us from us high quality and we hold ourselves to high standards. And we will deliver.
Support
Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.
George Abitbol* doesn’t feel well. Christmas is approaching at a fast pace and the gift he ordered days ago for his girlfriend didn’t find its way to his mailbox yet. He checked it out three times today and save for some spam catalogues on how to take care of his handsome silhouette, nothing resembling a gift showed up.
Picture by Saâd Kadhi
He tried to call the French parcel service to know the whereabouts of the luxurious, limited version of the organic sweet potato chips his lovely Jacqueline* likes so much, which sells for four times the regular price (to bear the cost of the enhanced packaging, certainly), but he couldn’t get hold of a living soul all day long. When he placed the order, the delivery was supposed to be lightning fast. It turned out to be a false promise.
With a sinking heart, he climbs back the stairs leading to his apartment, fetches his laptop and sits on his club chair. With his headphones on, immersed in the wonderful jazz of Christian Scott, he wanders randomly through online shopping sites trying to make out his mind on what other presents he could get for his dear Jacqueline, in time for Christmas.
In the middle of the track called Encryption, featuring the uncanny Elena Pinderhugues on flute, a Twitter notification resonates in his ears. He checks it out and learn that TheHive Chefs, as true and elegant gentlemen, have published a new training VM for Cerana 0.3 (a.k.a. TheHive 3.0.3), including Cortex 1.1.4 and the latest set of Cortex-Analyzers.
George loves bees in all shapes and forms, including digital ones so he swiftly downloads the new VM and as the cautious person he is, he verifies the file’s SHA256 hash: 86a87b70627e8db672c57cb57821461f2564ae9b8087cc22fdd1e7a599c16aed. Wonderful! Everything checks out beautifully. He then imports the file in his VM software, starts the virtual machine and logs in as thehive then types in thehive1234 when asked for the password.
He thoroughly reads the documentation to configure various analyzers and integrate his favourite Security Incident Response Platform with MISP. A few minutes later, his VM is ready for prime time and he starts playing with the new multi-source dashboards and interacting with fellow analysts on Gitter.
And he totally forgets about Jacqueline’s gift.
(*) Any resemblance to real and actual names is purely coincidental.
Correction: Dec 23, 2017 An earlier version of this post was referring to a previous training VM that included Cerana 0.2, a version affected by a privilege escalation vulnerability which was corrected in Cerana 0.3. Some typos were corrected as well.
TheHive Chefs are happy to announce the immediate availability of their latest recipe: Cerana 0.2 (a.k.a. TheHive 3.0.2).
While TheHive 3.0.0 brought you dynamic dashboards among other niceties, the latest (and, of course, the greatest) version of your Mom’s favourite Security Incident Response Platform fixes a bug spotted by our longtime supporter Megan Roddie (merci !). Indeed, Nabil was running low on coffee so he didn’t make the necessary changes to support the new sighted toggle introduced by Cerana for file observables.
Last but not least, we decided to make dynamic dashboards even more powerful. You can now create new graphs that supportmultiple series from multiple entities (or sources).
Multi-line Dashboards Example — Number of IOCs imported from MISP vs. those imported from other sources
As Christmas is approaching, go ahead and play with dynamic dashboards to impress your management as soon as 2018 rears its head or truly drive your CTI and DFIR activities and plan well ahead how you should improve automation or collaboration (or beg for additional headcount).
Multi-line Dashboards Example — How to create one
Ain’t that nifty? Who said bees aren’t nice? Joyeux Noël !
Feeling Generous? Donate!
As you know, we are a FOSS project and donations are always welcome to make our products even better for the community.
All donations go to Creative Source, the non-profit organization we have created, and we will use them to improve TheHive, Cortex & Hippocampe but also to develop (even better) integrations with other FOSS solutions such as MISP.
So if you are feeling generous, contact us at support@thehive-project.org. Of course the funds may also be used to keep Nabil happy by providing a steady flow of caffeine. 😉
Creative Source can also provide so-called professional, entreprise-grade support, help integrating the products, train your analysts before they drain or assist you in specific areas such as developing in-house analyzers for Cortex.
Download & Get Down to Work
If you have an existing installation of TheHive, please follow the migration guide.
If you are performing a fresh installation, read the installation guide corresponding to your needs and enjoy. Please note that you can install TheHive using an RPM or DEB package, use Docker, install it from a binary or build it from sources.
Support
Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.
Two days after the release of Mellifera 13.2 (TheHive 2.13.2), we have updated the training VM with this version. You can download it from the following location:
Something does not work as expected? No worries, we got you covered. Please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.
Following the release of Mellifera 13 last week, some users reported problems getting the platform working correctly. They couldn’t browse a case’s tasks. TheHive Chefs reproduced the bug and corrected swiftly in Mellifera 13.1 (TheHive 2.13.1), which is now available. Please note that the identified bug happens only when you haven’t upgraded TheHive from an earlier version.
Is ES 2.x still supported?
Mellifera 13 introduced the support of Elasticsearch 5.x and has been thoroughly tested with version 5.5 (5.6 should be probably work just fine). Given the numerous changes between ES 2.x and ES 5.x, we do not support both versions. Hence, and starting from Mellifera 13, only ES 5.x is supported.
Download & Get Down to Work
If you have an existing installation of TheHive, please follow the migration guide. This is paramount to ensure a good transition from earlier versions. You have been warned.
If you are performing a fresh installation, read the installation guide corresponding to your needs and enjoy. Please note that you can install TheHive using an RPM or DEB package, use Docker, install it from a binary or build it from sources.
Support
Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.
TheHive Project French Chefs are very happy to announce Mellifera 12, a brand new, all shiny, major version of TheHive.
This MAGA (Make Analysis Great Again) edition of your (soon to be?) favorite Security Incident Response Platform (SIRP) contains an awful lot of changes and improvements. While you can read the full changelog while waiting in line for your burger at HopDoddy or for your turn for the latest Disney attraction, we’d like to concentrate on a few features that would make you enjoy Digital Forensics & Incident Response like never before 🙂
Go Short or Go North
The Observables tab can now display the short (a.k.a. mini) reports produced by analyzers. Whenever you run an analysis (or many), the mini-reports will be shown as soon as the corresponding analyzer jobs have successfully finished. That way, you no longer have to click on each observable to access the short report.
Short Reports shown on the Observables Tab
In fact, you don’t even have to click on the observable to access the long reports. You must simply click on the corresponding short report and the long one will be displayed on top of the observables tab as shown in the following screencast. Ain’t that nifty?
A Single Click on the Short Report Shows the Long One
The short reports have been also improved to follow a taxonomy. To get to this stage, we had to review all 24 analyzers and their flavors, add new functionality to the CortexUtils Python library and improve the analyzers to add a summary section to their JSON output which Mellifera 12 interprets and displays according to a color code as described in our previous post. Please make sure to read it as it contains important information on how to update your cortexutils version and the analyzers as well as the report templates.
Is This Alert New or What?
Mellifera 12 introduces an important feature pertaining to alerts. To put it simply, whenever you receive a new alert from MISP, email, SIEM or any other source that you have connected with TheHive, the alert preview page will tell you if there are similarities with existing cases and if so, Mellifera 12 will let you import the new alert in the existing case and any updates made to that alert (think of an ongoing MISP event) will be automatically added to the case.
Alert Preview Page with the new Similar cases Section
Template this, Template that
In addition to the ‘similarity’ feature outlined above, Mellifera 12 lets you choose the case template to use when importing a new alert instead of having to use only a specific case template per alert type/source.
Choose the Template You’d Like to Use to Import an Alert
Custom Fields
We heard our community and implemented a feature that was requested by several users: custom fields.
So you’d like to add a business impact to a specific type of cases? Or a set of TTPs? Or a Threat Actor? Or specify a Business Unit? No problem! Ask an admin to create a custom field, associate it with a case template and there you go.
Add a Custom Field
Unlike metrics, custom fields must not be filled to close a case. You can also supercharge a case with custom fields that have not been associated to a case template. We currently support four types of custom fields: strings, numbers, booleans and dates. And you can create lists of acceptable values to limit your analysts’ choices to legitimate data.
If you have an existing installation of TheHive, please follow the migration guide.
If you are performing a fresh installation, read the installation guide corresponding to your needs and enjoy. Please note that you can install TheHive using an RPM or DEB package, use Docker, install it from a binary or build it from sources.
Support
Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.
TheHive, Cortex and MISP work nicely together and if you’ve read our June-Dec 17 roadmap post, the integration of our products with the de facto threat sharing platform will get better in a few months.
During the FIRST conference presentation we gave last week, we displayed a picture that we will use here to try to explain how these three open source and free products integrate with one another.
A Picture is Worth a Thousand Words…
TheHive
TheHive is a Security Incident Response Platform (SIRP). It can receive alerts from different sources (SIEM, IDS, email. etc.) via its REST API. This is where alert feeders come into play.
Alert Feeders
Think of an alert feeder as a specialized program which consumes a security event (SIEM alert, email report, IDS alert, and so on), parses it and outputs an alert that its sends to TheHive through TheHive4py, the Python library we provide to interact with TheHive’s REST API.
We do not supply such feeders but developing them should be straightforward. If not, let us know and we’ll do our best to help you out.
Alerts
Any alert sent to TheHive will show up in its Alerts pane. In addition to the sources mentioned above, new or updated MISP events will show up as well in that area if you configured TheHive to connect to one or several MISP instances. If so, TheHive will poll those MISP instance(s) at every interval looking for new or updated events. If there are any, TheHive will generate an alert which will end up in the Alerts pane.
The Alerts Pane
Alerts can be ignored, mark as read, previewed and imported. When an alert is imported, it becomes a case that needs to be investigated.
Cases
The Workflow that is at the Heart of TheHive
A case can be generated from an alert or created from scratch. It is subdivided into tasks (think identification, containment, eradication, check proxy logs, and so on) and observables (IP addresses, hashes, email addresses, domain names, URLs…). When analysts are working on tasks, they add logs as they go. In TheHive’s terminology, logs are text entries which may contain attachments to help analysts record what they have been doing. Logs can be written using Markdown or a rich-text editor.
Case Templates
You don’t need to add the same tasks over and over when working on cases belonging to a given category (DDoS, Malspam, APT, …). You can create custom templates to which you add tasks as shown below. This is very useful when you are dealing with alerts so that when you import them, you can select which case template you’d like to apply and there you go!
A Sample Case Template
Observables
Observables can be tagged, flagged as IOCs, and analyzed. When the investigation is well in progress or completed, you may want to share the resulting IOCs or a subset of those with partners and peers. TheHive will support the ability to export that data to MISP in September 2017. Until then, you can still export your IOCs as text, CSV or as a MISP-compatible format that you can use to add them to your MISP instance using the freetext editor. TheHive can export IOCs/observables in protected (hxxps://www[.]somewhere[.]com/) or unprotected mode.
Every observable must have a TLP (Traffic Light Protocol) level. By default, any added observable is considered TLP:AMBER. Please note that the TLP is taken into account by some analyzers. Wait! Analyzers?
Cortex
Cortex is our standalone analysis engine and a perfect companion for TheHive and MISP. Analysts can use it to analyze observables using its Web UI, in which case they can be submitted only one at a time. The Web UI should really be limited to quick assessments of observables before creating a case in TheHive (or in an alternate SIRP). The power of Cortex really comes into play when you use its REST API. TheHive speaks natively to Cortex (as MISP does). Moreover, TheHive can leverage one or several Cortex servers.
Observable Page and List of Analyzers
Analyzers
As of this writing, Cortex has 23 analyzers which come in a total of 39 flavors and more will be available soon.
An analyzer can be written in any programming language supported by Linux though all of our current analyzers are written in Python. This is because we provide a Python library called Cortexutils which contains a set of utility classes that make it easier to write an analyzer in Python.
Flavors
Analyzers such as VirusTotal, PassiveTotal or DomainTools can provide different analysis services. Let’s take VirusTotal as an example. You can scan a file or URL. That’s one flavor. You can also obtain the latest available report on VirusTotal.com for a file, hash, domain or IP address. That’s a second flavor. So the VirusTotal analyzer has two flavors.
How about PassiveTotal? It has 8 flavors: unique resolutions lookup, SSL certificate history lookup, malware lookup, passive DNS lookup, data enrichment lookup, SSL certificate details lookup, OSINT lookup and WHOIS data lookup.
The MISP Search Analyzer
At this point, we need to mention a special analyzer that may create some confusion if not understood correctly: the MISP Search analyzer. Thanks to it, Cortex has the ability to search observables within a MISP instance as represented by the arrow that goes from the Analyzers to MISP.
Search for MISP Events Containing a Given Observable
When an observable is found in an event, Cortex will return the number of records found (i.e. the number of events where the observable has been found) and a list of links to those events with additional data.
Searching for a Hash Using the MISP Search Analyzer from the Cortex Web UIThe Same Search Conducted from TheHive: Long ReportMini-Report
The current version of the MISP Search analyzer can only search within a single MISP instance but in the near future, it will be able to support multiple ones.
MISP Expansion Modules
Besides its own analyzers (which include MISP Search described above), Cortex can also invoke MISP expansion modules. These are normally used by MISP to enrich attributes within events but Cortex can also take advantage of them to analyze observables.
There is some overlap between the native Cortex analyzers and MISP expansion modules. When choosing between a native analyzer or an expansion module, we highly recommend you select the former. The expansion modules are deactivated in the default Cortex configuration.
Jobs
When you submit an observable for analysis, Cortex will create a job and, if successful, it will generate an analysis report in JSON format. TheHive has the ability to parse those results and present them in a human-friendly fashion thanks to report templates we offer for free. So when you’ll submit an observable to Cortex from TheHive, you’ll get back a short (or mini) report and a long one. The first can be thought of as a really tiny Exec Analyst Summary while the second provides more insight and details.
Calling Cortex from MISP
In addition to the expansion modules we have just mentioned, MISP 2.4.73 and up can enrich attributes using Cortex analyzers. The configuration is pretty straightforward. So if all you are concerned about is threat intelligence and sharing, you may augment your visibility into a given threat represented as a MISP event by leveraging all current 23 Cortex analyzers and any future ones.
Conclusion
TheHive, Cortex and MISP are three open source and free products that can highly aid you combat threats and keep the ‘monsters’ at bay.
TheHive, as a SIRP, allows you to investigate security incident swiftly in a collaborative manner. Several analysts can work simultaneously on tasks & cases . While cases can be created from scratch, TheHive can receive alerts from different sources thanks to alert feeders which consume security events generated by multiple sources and feed them into TheHive using TheHive4py Python library. TheHive can also sync to one or several MISP instances to receive new and updated events which will appear in the alert pane with all the other alerts generated by other sources. Analysts can then preview new alerts to decide whether they need to be acted upon. If so, they can transform them into investigation cases using templates.
To analyze the observables collected in the course of an investigation and/or imported from a MISP event, TheHive can rely on one or several Cortex analysis engines. Cortex is another standalone product that we have developed which sole purpose is to allow you to analyze observables at scale thanks to its large number of analyzers, MISP expansion modules and any analyzer you might have developed on the side. Cortex has a REST API that can be used to empower other security products such as ‘analytics’ software, alternate SIRPs or MISP.
The highly popular threat sharing platform can indeed enrich attributes thanks to Cortex as it has a native integration with it. And in a few months, you will also be able to export cases from TheHive as MISP events that you can share with peers and partners.
If you do share, you do care about our collective mission to defend thedigital assets that are under our watch from harm. So let us fight together as one.