Some of our die-hard fans noticed that we silently released TheHive 3.3.0 a few days ago, after six release candidates. Well. Silently won’t be the right word to use in this case as we are drowning under work and feature requests and we sometimes postpone communication in favour of getting true real work done.
So, without any further ado, we are happy to announce the official availability of our latest (and of course greatest) release of the most-advanced, next-gen, HI (Human Intelligence), gluten-free, (add here any keyword that you fancy to help us get the Gartner attention and land in the Magic Quadrant™), free and open source Security Incident Response Platform Security Orchestration Automation & Response Platform.
As stated earlier, TheHive 3.3.0 went through the largest number of release candidates to date in order to ensure it contains more features than bugs (or unexpected functionality as our dear Nabil call them sometimes).
TheHive and Cortex are a huge success. According to our estimates, there are about a hundred organisations of different sizes and locations using or testing them. And as the number of users grows, so does the number of features, professional service and support requests.
We tried finding a solution to keep TheHive Project as healthy as possible. So we created Creative Source, a nonprofit organisation (NPO), in the hope that we could leverage it to hire more developers thanks to the generous donations of our large user community. Sadly, not everyone in this world is generous and altruistic. At the end, all but one company (yes, exactly one) trusted us enough to make a donation and get tailored services for its needs in return. Most of the others either did not reply to our proposals or explained that their procurement process does not accommodate working with NPOs.
As we informed you a few weeks ago, some members of our core team are finalising an alternative option to ensure not only the viability of TheHive and Cortex as FOSS products on the long run but the ability to provide professional training, support, and services without making highly bureaucratic, think-in-the-box-but-never-outside procurement departments freak out.
TheHive Project’s code Chefs, dressed in their outright haute cuisine outfit, including the traditional toque blanche, have been quite busy lately, working on dockerizing all the Cortex analyzers (more on this later in an upcoming post), and doing tedious work to prepare the replacement of Elasticsearch by a GraphDB which will help us finally release much-awaited features such as multi-tenancy, delayed for way too many months (yeah, yeah, don’t chastise them but feel free to help them). In the meantime, they found enough bandwidth to release a new major version of TheHive.
Version 3.3, currently a release candidate includes several bug fixes and many new features as outlined below. Please note that TheHive 3.3-RC2 is beta software. As all our other release candidates, you can grab it from the pre-release, beta repositories. As usual, we would truly appreciate your help making it a great stable release by testing it as thoroughly as possible and reporting back any bugs or issues you encounter so we can address them before the final release.
TheHive 3.3-RC1 was very short-lived. Few hours after its release, and thanks to Chris (a.k.a. crackytsi on GitHub), Thomas Franco, our back-end mastermind, discovered an issue with the Debian 8 and Debian 9 packages.
#836: add a new exportCaseTags parameter to the MISP configuration section. If set to true, all the tags associated with a case will be exported along with it to MISP.
#861: add support for Java higher than 8, such as OpenJDK 11.
#271: bulk merge alerts into a case. Select multiple alerts at once and create a single case out of them or merge them into an existing case using its ID.
#824: add ability to sort alerts by reference, status, type, source…
#826: when previewing an alert, there are sometimes no overlap with an existing case. However, an analyst might already know, thanks to HI (Human Intelligence), that the alert should be merged into a specific case. This is now possible thanks to a new button.
#769: improve case template selection for case creation. If you have defined a large set of case templates, you will be able to sort/filter to find the case you want to use when creating a New Case.
#657: add observable tags auto-completion. Contributed by Tyler Chong (Thanks!).
#864: do not return a session cookie when making an API call.
#856: there was a bug where after a followed alert PATCH, if the alert has already been promoted to a case, the case is not updated. Now, if the alert has follow=true, if it gets updated, its status is set to Updated and the related case is updated too.
#845: assigned but unstarted tasks were not showing up in My Tasks.
#844: enable user account locking through the Delete API endpoint.
A few months ago, we released Synapse, our first “meta” alert feeder for TheHive in order to lower the burden of highly repetitive tasks on incident handlers. Thanks to a scalable and modular design, Synapse aims to make incident response easier by automating some of its tedious parts.
The first step towards this challenge was based on the integration of Microsoft Exchange with TheHive in Synapse 1.0.0. This way, user notifications pertaining to suspicious emails can be easily consumed and acted upon in TheHive.
Today, we are releasing version 1.1.0 which goes further by adding support for Microsoft Exchange O365 and the IBM QRadar SIEM.
Theoretically, Exchange O365 was supposed to be functional in Synapse 1.0.0.
However, since we did not have an O365 account we could not fully test that feature. Thankfully with the help of one of our users we managed to solve a bug and finally validate the Exchange O365 integration.
For more details about Exchange and TheHive, have a lookhere.
IBM QRadar SIEM
Members of TheHive’s Core Team have practical experience with QRadar and we decided to make good use of it to the benefit of our fellow analysts.
With the Community Edition of QRadar in one hand and an instance of TheHive in the other, we managed to create alerts in TheHive out of QRadar offenses. Furthermore, when a case or alert related to a QRadar offense is closed in TheHive, it also closes it in QRadar automatically.
For more details about QRadar and TheHive, have a look here. Alternatively, you may also want to consider Pierre Barlet’s qradar2thehivescript.
Send your Ideas our Way
With this 1.1.0 release, the list of integrated products with TheHive goes up to three: Exchange, Exchange O365 and QRadar. However, we don’t really have a plan regarding the next candidate for integration so tweet us at @TheHive_Project and tell us what you want!
Paris, France. The Sun is shining on the city of lights and temperatures are quite high, even for the summer season. Life is good. As a matter of fact, life is excellent.
TheHive Master Cooks are about to go on vacation for a few weeks. But before they pack up their Patagonia bags and leave the sandy beaches for those who enjoy them, preferring mountains, trails, walking and breathing fresh air with family and friends, they would like to make a significant contribution to help TheHive and Cortex users fight cyberattacks even better than they already do. And maybe convince those who don’t that free, open source software is not a joke or a geek fad.
We’d like to welcome to the stage our latest babies, which we are really proud of: TheHive 3.1 and Cortex 2.1, the new versions of the power duo which make digital forensics, incident response and, to an extent, cyber threat intelligence, better, faster, happier, regular exercising (well you know the Radiohead song so we’ll let you continue singing along) since early 2017.
While our project might seem very young, it is not. We’ve been working steadily on TheHive, using it (i.e. eating our own dog food) since early 2014 before releasing it at the end of 2016 once we were satisfied with it, as a token of gratitude to a community that helped us due our jobs in various ways. We then extracted what has become Cortex from its core to ship it as a separate product in February 2017. And we kept improving them at a steady piece for the collective benefit of incident responders, forensicators and threat analysts. And adoption has been rather spectacular. Thanks to all of our users for their love and support!
We believe we are at a moment where people could not brush us off anymore as amateurs. Try TheHive and Cortex, preferably with MISP and get a taste of what professional, free and open source software can be.
TheHive 3.1 and Cortex 2.1 are feature-packed and we won’t be able to cover them all in detail in a single blog post. Rather, we’d like to concentrate on a few important ones.
Stable, Pre-release Channels and New Repositories
We’d like to point out that, in order to improve our release process and given the number of features that we added, TheHive 3.1 and Cortex 2.1 are release candidates at this stage. So we encourage you to test them and report back any bugs or issues you encounter so we can address them and make the final releases as rock-solid as possible.
We have now two release channels: a stable one that should be used for production systems and a pre-release channel that should be used to try the release candidates such as TheHive 3.1-RC1 and Cortex 2.1-RC1, the subjects of this blog post, and help us iron out bugs before adding them to the stable channel. Those who love living on the bleeding edge may be tempted by running the release candidates on their production environment given all the candy and icing we added. They are at liberty of doing so but we don’t want to hear anyone one whining about an RC that broke everything and asking for their money back grin.
In previous releases of TheHive, whenever you configured a MISP instance, it was used to import events from and export cases to. Starting from TheHive 3.1.0, we added a purpose to the configuration file. By default, any added MISP instance will be used for import and export (ImportAndExport). However you can configure it to be used for importing events only (ImportOnly) or exporting cases only (ExportOnly).
When an analyst attempts to update a MISP event on which the account used by TheHive to connect to the MISP instance is not part of the original author’s organization, previous versions of TheHive will display a you do not have permission to do that error produced by MISP. Starting from TheHive 3.1, analysts have the ability to create a MISP extended event.
Case tasks can now be associated with task groups. For example, you could create groups called Identification and Malware Analysis, Containment and Communication and add tasks to them. Of course, this new feature can be used when designing case templates as well.
Import Observables from Analyzer Output
If analyzers produce a set of artifacts in their output (which is the case of several existing ones), TheHive will give you the ability to select those artifacts very easily and add them to your case as observables.
ZIP File Upload
Austin Haigh contributed an important feature which will allow analysts to directly import password-protected ZIP files into a case. The code uses the supplied password when adding the archive to extract its contents and add them one by one to the existing set of observables. This is highly practical when you want to add suspicious files without risking an accidental click which would compromise your endpoint or having to unzip archives containing such files first then add them one by one to TheHive.
Revamped Search Page
The search page has been completely revamped as shown in the screenshot below:
You can now select your search scope (cases, tasks, observables, alerts, analyser reports a.k.a. jobs or even the audit logs), apply filters and search TheHive without having to resort to complex, mind numbing Lucene syntax.
Responders and PAP
Last but not least, TheHive and Cortex offer you response capabilities (i.e. perform an action depending on the context) thanks to a new breed of programs called … wait for it … wait for it … responders. TADA!
Responders are very similar to analyzers. In fact we’ve taken the concept and extended it to apply to different elements in TheHive: alerts, cases, tasks, task logs, and observables of course.
You can reuse almost the same principles that apply to analyzers to write your own responders and if you are feeling generous, contribute them to the community. To give you a head start, we published a sample Mailer responder which, when customized for your environment, should allow you to send emails to inform your fellow analysts that a case has been created and that their help is required. Another example could be the ability to respond to a suspicious email report from a user, which is displayed as an alert, that they can safely ignore the corresponding email.
Like an analyzer, a responder can have two or more service interaction files (or flavors) to allow it to perform different actions. For example, a Mailer responder can send messages using several body templates.
Thanks to our long-time friend Andras Iklody from MISP Project who brought that to our attention, responders (and analyzers starting from Cortex 2.1) support PAP, the Permissible Actions Protocol.
Running into Troubles?
Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at firstname.lastname@example.org. We are here to help.
When we’re not busy cooking new features, we go back to the trenches and face incidents like many of our fellow analysts who read our publications and use our tools. To do so, we swap our chef toques for firefighter helmets, not only because such shiny headwear is cool, but mainly because incident response (IR) is, at its very heart, firefighting (minus all the dangerous stuff).
If you think about it, when handling incidents you can see everything from cats in trees (spam) to major fire (APT). Thankfully, there are more cats to bring down than fire to extinguish. That being said, a big herd of cats could be a serious threat to your organization, to your mental health or both.
We tend to forget that incident handlers are humans, not robots. Unlike our metal cyberfriends, we need diversity. We can’t risk insanity like Charlie Chaplin in Modern Times if we can avoid it. Unfortunately IR can be highly repetitive, especially if you only have cats to deal with.
Some could say ‘Nah, this is minor, nothing critical here’ but at some point, an analyst brainwashed by the same tasks again and again will be led to fault. In the worst case scenario, one could see an alert and immediately categorize it as false positive without any further consideration. Because ‘this alert is always a false positive’, until the day it is not…
Automation, a Solution?
Intuitively, we look in the direction of automation in order to minimize what we call ‘zombie’ tasks: highly repetitive and brainless tasks that need to be done. We believe that doing so will allow incident handlers to focus on the analysis and not on the tedious side of IR. Ultimately, we hope it will keep analysts stimulated and in a state of alert. Also, it should reduce time and effort spent on the low-hanging fruits.
One of the most dreary tasks in our opinion is to record the context around an incident.
What is the problem? When did it happen? What’s the origin? Who are the victims? How many are there? Answers to these questions let you have an overview of what is happening and are valuable to correlate incidents. So it is worth taking some minutes to add this information to your case. Sadly, most of the time it will look like a succession of ‘Ctrl+C; Alt+Tab; Ctrl+V’ from your incident source to TheHive. Exactly the kind of tasks we want to forego.
Having identified the threat that apathetic analysts pose, the root cause (highly repetitive tasks) and a solution (automate the recording of incident context), the question of the implementation has been raised.
The first challenge to solve is the number of incident sources. Almost everything can trigger an incident: a firewall, an IDS, antivirus, SIEM, users, etc… So the application must be designed to accept several sources and must permit to easily integrate new ones. And instead of having to configure multiple alert feeders to supply alerts to TheHive, we would have only one. To some extent, it can be assimilated to a meta feeder.
And if the application works as intended, we still have a second challenge. Let’s say you, dear reader, and ourselves use the galaxy renowned Stargazer IDS. Maybe you’d like to include the full packet capture in the case but we wouldn’t. Using the same product doesn’t mean using it the same way. So we have a variety of sources and for each source, we have a variety of configurations and workflows. Hence any app we design needs to accept multiple configurations and workflows for any given source.
Finally: the third challenge. We want to make the most out of TheHive. Creating cases, creating alerts, assigning cases, adding logs, adding observables … all those actions are not an option.
After several trials and failures, we came up with Synapse. Basically it is a Python 3app which sits between TheHive and your incident sources:
To solve the first and third challenge, we rely on connectors. A connector is a Python object dedicated to interact with a security device. In the picture above, you can see the Exchange Connector and TheHive Connector. To extend the number of sources, you just have to develop the connector that corresponds to your device.
Regarding the second challenge, we rely on workflows. Workflows are python scripts who use connectors to automate repetitive tasks when tracking a case. Not happy with the current workflow? Develop your own using the connectors.
At this point, you probably wonder why there’s an API in the picture above. Well, the API is the link between the user and the workflows. By hitting a specific endpoint of the Synapse’s API, the corresponding workflow will be launched. That way the user can choose what to launch, especially if they are only interested in a particular workflow. Moreover, using an API allows us to listen to TheHive’s real-time stream and initiate some actions like closing a QRadar offense when the related case is solved.
At the moment, Synapse includes the Exchange connector and the associated Ews2Case workflow. The workflow features:
Case creation from emails
Adding email bodies to task logs
Adding email replies to the case
Adding email attachments as observables
And of course, everything is done to minimize the number of clicks! Check the workflow documentation to understand how it works under the hood.
We’re still working on the QRadar connector and the associated workflows but if you can’t wait, have a look at the work done by the community like pierrebarlet’s script.
Check it Out
As usual, Synapse is an open source and free software released under the AGPL (Affero General Public License).
TheHive, Cortex and MISP work nicely together and if you’ve read our June-Dec 17 roadmap post, the integration of our products with the de facto threat sharing platform will get better in a few months.
During the FIRST conference presentation we gave last week, we displayed a picture that we will use here to try to explain how these three open source and free products integrate with one another.
TheHive is a Security Incident Response Platform (SIRP). It can receive alerts from different sources (SIEM, IDS, email. etc.) via its REST API. This is where alert feeders come into play.
Think of an alert feeder as a specialized program which consumes a security event (SIEM alert, email report, IDS alert, and so on), parses it and outputs an alert that its sends to TheHive through TheHive4py, the Python library we provide to interact with TheHive’s REST API.
We do not supply such feeders but developing them should be straightforward. If not, let us know and we’ll do our best to help you out.
Any alert sent to TheHive will show up in its Alerts pane. In addition to the sources mentioned above, new or updated MISP events will show up as well in that area if you configured TheHive to connect to one or several MISP instances. If so, TheHive will poll those MISP instance(s) at every interval looking for new or updated events. If there are any, TheHive will generate an alert which will end up in the Alerts pane.
Alerts can be ignored, mark as read, previewed and imported. When an alert is imported, it becomes a case that needs to be investigated.
A case can be generated from an alert or created from scratch. It is subdivided into tasks (think identification, containment, eradication, check proxy logs, and so on) and observables (IP addresses, hashes, email addresses, domain names, URLs…). When analysts are working on tasks, they add logs as they go. In TheHive’s terminology, logs are text entries which may contain attachments to help analysts record what they have been doing. Logs can be written using Markdown or a rich-text editor.
You don’t need to add the same tasks over and over when working on cases belonging to a given category (DDoS, Malspam, APT, …). You can create custom templates to which you add tasks as shown below. This is very useful when you are dealing with alerts so that when you import them, you can select which case template you’d like to apply and there you go!
Observables can be tagged, flagged as IOCs, and analyzed. When the investigation is well in progress or completed, you may want to share the resulting IOCs or a subset of those with partners and peers. TheHive will support the ability to export that data to MISP in September 2017. Until then, you can still export your IOCs as text, CSV or as a MISP-compatible format that you can use to add them to your MISP instance using the freetext editor. TheHive can export IOCs/observables in protected (hxxps://www[.]somewhere[.]com/) or unprotected mode.
Every observable must have a TLP (Traffic Light Protocol) level. By default, any added observable is considered TLP:AMBER. Please note that the TLP is taken into account by some analyzers. Wait! Analyzers?
Cortex is our standalone analysis engine and a perfect companion for TheHive and MISP. Analysts can use it to analyze observables using its Web UI, in which case they can be submitted only one at a time. The Web UI should really be limited to quick assessments of observables before creating a case in TheHive (or in an alternate SIRP). The power of Cortex really comes into play when you use its REST API. TheHive speaks natively to Cortex (as MISP does). Moreover, TheHive can leverage one or several Cortex servers.
An analyzer can be written in any programming language supported by Linux though all of our current analyzers are written in Python. This is because we provide a Python library called Cortexutils which contains a set of utility classes that make it easier to write an analyzer in Python.
Analyzers such as VirusTotal, PassiveTotal or DomainTools can provide different analysis services. Let’s take VirusTotal as an example. You can scan a file or URL. That’s one flavor. You can also obtain the latest available report on VirusTotal.com for a file, hash, domain or IP address. That’s a second flavor. So the VirusTotal analyzer has two flavors.
How about PassiveTotal? It has 8 flavors: unique resolutions lookup, SSL certificate history lookup, malware lookup, passive DNS lookup, data enrichment lookup, SSL certificate details lookup, OSINT lookup and WHOIS data lookup.
The MISP Search Analyzer
At this point, we need to mention a special analyzer that may create some confusion if not understood correctly: the MISP Search analyzer. Thanks to it, Cortex has the ability to search observables within a MISP instance as represented by the arrow that goes from the Analyzers to MISP.
When an observable is found in an event, Cortex will return the number of records found (i.e. the number of events where the observable has been found) and a list of links to those events with additional data.
The current version of the MISP Search analyzer can only search within a single MISP instance but in the near future, it will be able to support multiple ones.
MISP Expansion Modules
Besides its own analyzers (which include MISP Search described above), Cortex can also invoke MISP expansion modules. These are normally used by MISP to enrich attributes within events but Cortex can also take advantage of them to analyze observables.
There is some overlap between the native Cortex analyzers and MISP expansion modules. When choosing between a native analyzer or an expansion module, we highly recommend you select the former. The expansion modules are deactivated in the default Cortex configuration.
When you submit an observable for analysis, Cortex will create a job and, if successful, it will generate an analysis report in JSON format. TheHive has the ability to parse those results and present them in a human-friendly fashion thanks to report templates we offer for free. So when you’ll submit an observable to Cortex from TheHive, you’ll get back a short (or mini) report and a long one. The first can be thought of as a really tiny Exec Analyst Summary while the second provides more insight and details.
Calling Cortex from MISP
In addition to the expansion modules we have just mentioned, MISP 2.4.73 and up can enrich attributes using Cortex analyzers. The configuration is pretty straightforward. So if all you are concerned about is threat intelligence and sharing, you may augment your visibility into a given threat represented as a MISP event by leveraging all current 23 Cortex analyzers and any future ones.
TheHive, Cortex and MISP are three open source and free products that can highly aid you combat threats and keep the ‘monsters’ at bay.
TheHive, as a SIRP, allows you to investigate security incident swiftly in a collaborative manner. Several analysts can work simultaneously on tasks & cases . While cases can be created from scratch, TheHive can receive alerts from different sources thanks to alert feeders which consume security events generated by multiple sources and feed them into TheHive using TheHive4py Python library. TheHive can also sync to one or several MISP instances to receive new and updated events which will appear in the alert pane with all the other alerts generated by other sources. Analysts can then preview new alerts to decide whether they need to be acted upon. If so, they can transform them into investigation cases using templates.
To analyze the observables collected in the course of an investigation and/or imported from a MISP event, TheHive can rely on one or several Cortex analysis engines. Cortex is another standalone product that we have developed which sole purpose is to allow you to analyze observables at scale thanks to its large number of analyzers, MISP expansion modules and any analyzer you might have developed on the side. Cortex has a REST API that can be used to empower other security products such as ‘analytics’ software, alternate SIRPs or MISP.
The highly popular threat sharing platform can indeed enrich attributes thanks to Cortex as it has a native integration with it. And in a few months, you will also be able to export cases from TheHive as MISP events that you can share with peers and partners.
If you do share, you do care about our collective mission to defend thedigital assets that are under our watch from harm. So let us fight together as one.