Moments ago, we have announced the release of Mellifera 13, TheHive4py 1.3.0, and Cortex4py. And since we don’t want to leave you wanting for more fun time, you may want to schedule as well a Cortex update shall you need it 😉
Group ownership in Docker image prevents running on OpenShift #42
Fixed Bugs
Cortex removes the input details from failure reports #38
Display a error notification on analyzer start fail #39
Download & Get Down to Work
To update your current Cortex installation, follow the instructions of the installation guide. Before doing so, you may want to save the job reports that were not executed via TheHive. Cortex 1 has no persistence and restarting the service will wipe out any existing reports.
Please note that you can install Cortex using an RPM or DEB package, deploy it using an Ansible script, use Docker, install it from a binary or build it from sources.
Support
Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.
Following popular demand, the chefs at TheHive Project‘s code kitchen are happy to announce the immediate availability of Cortex4py.
What Is It?
Cortex4py is a Python API client for Cortex, a powerful observable analysis engine where observables such as IP and email addresses, URLs, domain names, files or hashes can be analyzed one by one using a Web interface or en masse through the API.
Cortex4py allows analysts to automate these operations and submit observables in bulk mode through the Cortex REST API from alternative SIRP platforms (TheHive has native support for one or multiple Cortex instances) and custom scripts.
Use It
To install the client, use PIP:
$ sudo pip install cortex4py
How Much Does it Cost?
Cortex4py is released under an AGPL license as all the other products we publish to help the IR community fight the good fight. So apart from the effort it’ll cost you to install and use, the price of our software is nada, zero, rien. But if you are willing to contribute one way or another, do not hesitate to drop us an email at support@thehive-project.org or contact us via Twitter.
Support
Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.
It’s time for us to rest. We are closing* TheHive’s code kitchen for most of August 2017 to enjoy all what France has to offer: wine (obviously), literature, beaches, mountains, gorgeous food, and a unique savoir vivre.
So keep calm and eat croissants.
Source : Pinterest
See you back soon. Au revoir ! —
(*) Bear with us as we will be particularly slow to respond to support and feature requests as well as Gitter queries.
While many are enjoying the summer holidays, the busy bees of TheHive Project have been working hard lately to develop new Cortex analyzers and review few of those submitted by our growing and thriving user community, bringing the grand total to 27. Yes, you read that right. Cortex can leverage 27 analyzers to help you analyze observables very simply in many different ways.
The latest update to the Cortex-analyzers repository contains 3 new analyzers: Yeti, Cuckoo Sandbox and WOT, described below. And your first step to benefit from them should consist of refreshing your master working copy on your Cortex instance:
$ cd where/your/analyzers/are
$ git pull master
Yeti
YETI is a FOSS platform meant to organize observables, indicators of compromise, TTPs, and knowledge on threats in a single, unified repository. It is mainly developed by fellow APT busters Thomas Chopitea and Gael Muller (who said France doesn’t produce good software?).
The new Cortex analyzer for this platform lets you make API calls to YETI and retrieve all available information pertaining to a domain, a fully qualified domain name, an IP address, a URL or a hash.
To be able to use the analyzer edit the Cortex configuration file (/etc/cortex/application.conf) and add the following lines:
Yeti {
# URL of the Yeti server: example: http://120.0.0.1:5000
url = ""
}
When called from TheHive, the following output is produced:
TheHive: YETI analyzer — Short and Long Report Samples
CuckooSandox
The Cuckoo Sandbox analyzer has been submitted by Andrea Garavaglia (Thanks!) and you can use it to analyze files and URLs with Cuckoo Sandbox.
By default, we chose to limit analysis to TLP:WHITE and TLP:GREEN observables for OPSEC reasons, in case your Cuckoo server provides Internet access to potentially harmful files. If you want to use it with TLP:AMBER or TLP:RED observables, edit CuckooSanbox_File_analysis.json orCuckooSanbox_URL_analysis.json and change the max_tlp parameter to 2 or 3.
To use the analyzer, edit the Cortex configuration file and add the following lines:
CuckooSandbox {
url = “http://mycuckoosandbox”
}
When called from TheHive, the following output is produced:
TheHive: Cuckoo Sandbox Analyzer — Short and Long Report Samples
WOT
The WOT analyzer was also submitted by Andrea Garavaglia (kudos!). Use it to check reputation of a given domain on the Web of Trust service. It takes domains and FQDNs as input.
An API key is needed to use this service, and has to be added in the Cortex configuration file:
WOT {
# API key of the Web of Trust account
key=“”
}
When called from TheHive, the following output is produced:
TheHive: WOT Analyzer — Short and Long Report Samples
Rejoice folks! You can now play with TheHive & Cortex thanks to the test VM we created. It includes Mellifera 12, the latest major version of TheHive, Cortex 1.1.3, the latest Cortex analyzers with all dependencies and ElasticSearch installed on top of Ubuntu 16.04 with Oracle JRE 8.
The test VM is intended to be used… well… for testing or training purposes. We strongly encourage you to refrain from using it for production.
Get It
You can download the VM from the following location:
click on Import templates button and select the downloaded package
Plug it with MISP
The test VM does not contain a MISP instance and none is configured in TheHive’s configuration file. To play with MISP, you may want to use the VM our good friends at CIRCL provide. Once you’ve downloaded it or if you have an existing instance, edit /etc/thehive/application.conf and follow the configuration guide.
Restart or Go Mad
After each modification of /etc/thehive/application.conf do not forget to restart the service:
$ sudo service thehive restart
Troubles?
TheHive service logs are located in /var/log/thehive/application.log.
Configure Cortex
All available analyzers are installed with their dependencies, but none is configured. To configure analyzers, edit /etc/cortex/application.conf and follow the configuration guide.
Restart or Go Mad
After each modification of /etc/cortex/application.conf do not forget to restart the service:
$ sudo service cortex restart
Troubles?
Cortex service logs are located in /var/log/cortex/application.log.
Need Help?
Something does not work as expected? No worries, we got you covered. Please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.
Correction: July 8, 2017
An earlier version of this post offered to download the VM from Dropbox but they suspended the associated link due to seemingly heavy traffic. The post was updated to replace the Dropbox link with a Google Drive one.
Ahead of the imminent release of Mellifera 12 (TheHive 2.12.0), a new, major (as in MAJOR) version of your (soon to be?) favorite Security Incident Response Platform, we’ve made rather significant changes to Cortex analyzers and released a new version of the CortexUtils Python library.
Adjust the Cortex configuration for the new MISP 2.0 analyzer and for Hippocampe as shown below if you are using these analyzers then import the corresponding report templates into TheHive:
click on Import templates button and select the downloaded package
CortexUtils 1.2.0
CortexUtils has been updated to include a new function called build_taxonomy() which is required for analyzers relying on the Python library we released to make their development development easier.
Mini-Reports in The Observable Tabs
Starting from Mellifera 12 (TheHive 2.12.0), mini-reports will be displayed in the observable tab in each case as soon as an analysis has been completed. Now analyzers compute their short/mini reports and put them in the summary section of their JSON output, ready for consumption. TheHive 2.12.0 and up will no longer create them on-the-fly.
Taxonomy
The mini-reports of all the analyzers have been updated to comply with a taxonomy that is similar to the one we were already using for VirusTotal: VT:Score="14/56”. A “maliciousness” level was already included in TheHive’s analyzer templates and we used a specific color to display each level. This level is now produced directly by the analyzers:
info / blue: the analyzer produced an information, and the short report is shown in blue color in TheHive.
safe / green : the analyzer did not find anything suspicious or the analyzed observable is safe (according to the analyzer). TheHive displays the short report in green color.
suspicious / orange : the analyzer found that the observable is either suspicious or warrants further investigation. The short report is orange colored in TheHive.
malicious / red : the analyzer found that the observable is malicious. The short report is displayed by TheHive in red color as show below:
The short report is built with the summary() function of an analyzer. The build_taxonomy() of cortexutils mentioned earlier should help building it.
MISP 2.0
The MISP analyzer has been updated to version 2.0 and includes new functionality submitted by our long-term contributor Nils Kuhnert from CERT-Bund (thanks a heap!). Unlike the previous version, v 2.0 will let you search for an observable in multiple MISP servers at the same time.
The analyzer accepts a truckload of datatypes as input. To make it work, install the pymispPython library. It should already have been installed if you are just updating your current analyzers. You will also have to change Cortex configuration file (application.conf) for this new version:
Important note: You have to adjust your existing configuration to match the one shown above. The certpath variable can be left blank if you are not using a self-signed certificate.
When called from TheHive, the following output is produced:
TheHive: MISP 2.0 Analyzer – Short and Long Report Samples
The short report will show the number of unique events found in all MISP servers while the long report will show information of each matching event in each MISP server.
CERTatPassiveDNS
The CERTatPassiveDNS analyzer is a new submission by Nils (thanks again). It lets you check the CERT.at PassiveDNS service for a given domain or hostname. It takes domains and FQDN as input.
Access to the CERT.at service is allowed to trusted partners only. If you think you qualify, please contact CERT.at. You do not need to add specific information into the Cortex configuration file to benefit from this analyzer as it calls the whois system command to perform the pDNS requests.
When called from TheHive, the following output is produced:
TheHive: CERTatPassiveDNS Analyzer – Short and Long Report Samples
Miscellaneous
The latest version of the Cortex-analyzers repository also include the following bug fixes and improvements:
#61: fix typos in two requirements.txt files – by Michael Salsone
#65: update the Joe Sandbox analyzer’s long report to support version 19.0.0
#67: fixed mistake in the FireHol analyzer – by Nils Kuhnert
#69: use http://server:port for Hippocampe instead of http://server:port/hippocampe/api/v1.0/
Running Into Trouble?
Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!
Correction: July 6, 2017
An earlier version of this post mispelled Nils Kuhnert’s last name.
TheHive, Cortex and MISP work nicely together and if you’ve read our June-Dec 17 roadmap post, the integration of our products with the de facto threat sharing platform will get better in a few months.
During the FIRST conference presentation we gave last week, we displayed a picture that we will use here to try to explain how these three open source and free products integrate with one another.
A Picture is Worth a Thousand Words…
TheHive
TheHive is a Security Incident Response Platform (SIRP). It can receive alerts from different sources (SIEM, IDS, email. etc.) via its REST API. This is where alert feeders come into play.
Alert Feeders
Think of an alert feeder as a specialized program which consumes a security event (SIEM alert, email report, IDS alert, and so on), parses it and outputs an alert that its sends to TheHive through TheHive4py, the Python library we provide to interact with TheHive’s REST API.
We do not supply such feeders but developing them should be straightforward. If not, let us know and we’ll do our best to help you out.
Alerts
Any alert sent to TheHive will show up in its Alerts pane. In addition to the sources mentioned above, new or updated MISP events will show up as well in that area if you configured TheHive to connect to one or several MISP instances. If so, TheHive will poll those MISP instance(s) at every interval looking for new or updated events. If there are any, TheHive will generate an alert which will end up in the Alerts pane.
The Alerts Pane
Alerts can be ignored, mark as read, previewed and imported. When an alert is imported, it becomes a case that needs to be investigated.
Cases
The Workflow that is at the Heart of TheHive
A case can be generated from an alert or created from scratch. It is subdivided into tasks (think identification, containment, eradication, check proxy logs, and so on) and observables (IP addresses, hashes, email addresses, domain names, URLs…). When analysts are working on tasks, they add logs as they go. In TheHive’s terminology, logs are text entries which may contain attachments to help analysts record what they have been doing. Logs can be written using Markdown or a rich-text editor.
Case Templates
You don’t need to add the same tasks over and over when working on cases belonging to a given category (DDoS, Malspam, APT, …). You can create custom templates to which you add tasks as shown below. This is very useful when you are dealing with alerts so that when you import them, you can select which case template you’d like to apply and there you go!
A Sample Case Template
Observables
Observables can be tagged, flagged as IOCs, and analyzed. When the investigation is well in progress or completed, you may want to share the resulting IOCs or a subset of those with partners and peers. TheHive will support the ability to export that data to MISP in September 2017. Until then, you can still export your IOCs as text, CSV or as a MISP-compatible format that you can use to add them to your MISP instance using the freetext editor. TheHive can export IOCs/observables in protected (hxxps://www[.]somewhere[.]com/) or unprotected mode.
Every observable must have a TLP (Traffic Light Protocol) level. By default, any added observable is considered TLP:AMBER. Please note that the TLP is taken into account by some analyzers. Wait! Analyzers?
Cortex
Cortex is our standalone analysis engine and a perfect companion for TheHive and MISP. Analysts can use it to analyze observables using its Web UI, in which case they can be submitted only one at a time. The Web UI should really be limited to quick assessments of observables before creating a case in TheHive (or in an alternate SIRP). The power of Cortex really comes into play when you use its REST API. TheHive speaks natively to Cortex (as MISP does). Moreover, TheHive can leverage one or several Cortex servers.
Observable Page and List of Analyzers
Analyzers
As of this writing, Cortex has 23 analyzers which come in a total of 39 flavors and more will be available soon.
An analyzer can be written in any programming language supported by Linux though all of our current analyzers are written in Python. This is because we provide a Python library called Cortexutils which contains a set of utility classes that make it easier to write an analyzer in Python.
Flavors
Analyzers such as VirusTotal, PassiveTotal or DomainTools can provide different analysis services. Let’s take VirusTotal as an example. You can scan a file or URL. That’s one flavor. You can also obtain the latest available report on VirusTotal.com for a file, hash, domain or IP address. That’s a second flavor. So the VirusTotal analyzer has two flavors.
How about PassiveTotal? It has 8 flavors: unique resolutions lookup, SSL certificate history lookup, malware lookup, passive DNS lookup, data enrichment lookup, SSL certificate details lookup, OSINT lookup and WHOIS data lookup.
The MISP Search Analyzer
At this point, we need to mention a special analyzer that may create some confusion if not understood correctly: the MISP Search analyzer. Thanks to it, Cortex has the ability to search observables within a MISP instance as represented by the arrow that goes from the Analyzers to MISP.
Search for MISP Events Containing a Given Observable
When an observable is found in an event, Cortex will return the number of records found (i.e. the number of events where the observable has been found) and a list of links to those events with additional data.
Searching for a Hash Using the MISP Search Analyzer from the Cortex Web UIThe Same Search Conducted from TheHive: Long ReportMini-Report
The current version of the MISP Search analyzer can only search within a single MISP instance but in the near future, it will be able to support multiple ones.
MISP Expansion Modules
Besides its own analyzers (which include MISP Search described above), Cortex can also invoke MISP expansion modules. These are normally used by MISP to enrich attributes within events but Cortex can also take advantage of them to analyze observables.
There is some overlap between the native Cortex analyzers and MISP expansion modules. When choosing between a native analyzer or an expansion module, we highly recommend you select the former. The expansion modules are deactivated in the default Cortex configuration.
Jobs
When you submit an observable for analysis, Cortex will create a job and, if successful, it will generate an analysis report in JSON format. TheHive has the ability to parse those results and present them in a human-friendly fashion thanks to report templates we offer for free. So when you’ll submit an observable to Cortex from TheHive, you’ll get back a short (or mini) report and a long one. The first can be thought of as a really tiny Exec Analyst Summary while the second provides more insight and details.
Calling Cortex from MISP
In addition to the expansion modules we have just mentioned, MISP 2.4.73 and up can enrich attributes using Cortex analyzers. The configuration is pretty straightforward. So if all you are concerned about is threat intelligence and sharing, you may augment your visibility into a given threat represented as a MISP event by leveraging all current 23 Cortex analyzers and any future ones.
Conclusion
TheHive, Cortex and MISP are three open source and free products that can highly aid you combat threats and keep the ‘monsters’ at bay.
TheHive, as a SIRP, allows you to investigate security incident swiftly in a collaborative manner. Several analysts can work simultaneously on tasks & cases . While cases can be created from scratch, TheHive can receive alerts from different sources thanks to alert feeders which consume security events generated by multiple sources and feed them into TheHive using TheHive4py Python library. TheHive can also sync to one or several MISP instances to receive new and updated events which will appear in the alert pane with all the other alerts generated by other sources. Analysts can then preview new alerts to decide whether they need to be acted upon. If so, they can transform them into investigation cases using templates.
To analyze the observables collected in the course of an investigation and/or imported from a MISP event, TheHive can rely on one or several Cortex analysis engines. Cortex is another standalone product that we have developed which sole purpose is to allow you to analyze observables at scale thanks to its large number of analyzers, MISP expansion modules and any analyzer you might have developed on the side. Cortex has a REST API that can be used to empower other security products such as ‘analytics’ software, alternate SIRPs or MISP.
The highly popular threat sharing platform can indeed enrich attributes thanks to Cortex as it has a native integration with it. And in a few months, you will also be able to export cases from TheHive as MISP events that you can share with peers and partners.
If you do share, you do care about our collective mission to defend thedigital assets that are under our watch from harm. So let us fight together as one.
Lo and behold, The Chefs behind TheHive Project have been pretty busy at the code kitchen, continuously improving some of your favorite recipes. After releasing Mellifera 11.3 and Cortex 1.1.3, here comes the turn of Cortexutils, the Python library containing utility classes for Cortex analyzers.
Version 1.1.1 of the library solves an encoding issue, in fact a regression introduced by the previous release. It also corrects a situation where error reports wouldn’t be generated in case the analyzer did not require configuration (we have a handful of such creatures in store).
You can grab the new Cortexutils release through PIP. To update your existing installation, please run the following command:
sudo pip install cortexutils --upgrade
Running Into Trouble?
Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!
Moments ago, we have announced the release of Mellifera 11.3. And since we don’t want to leave you wanting for more update er fun time, you may want to schedule as well a Cortex update shall you need it 😉
Cortex 1.1.3 is now available. This hotfix corrects the deb package to make it compatible with Ubuntu 16.04 without having to fiddle with OpenJDK. As we did with TheHive, we have repackaged the software to avoid grabbing OpenJDK 9 (which Cortex does not support) and force the installation of version 8. This version also corrects a cryptic error that might be thrown out by Cortex as a result of an improper interpretation of an analyzer failure.
Download & Get Down to Work
To update your current Cortex installation, follow the instructions of the installation guide. Before doing so, you may want to save the job reports that were not executed via TheHive. Cortex 1 has no persistence and restarting the service will wipe out any existing reports.
Please note that you can install Cortex using an RPM or DEB package, deploy it using an Ansible script, use Docker, install it from a binary or build it from sources.
Support
Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.
We are glad to announce a new version of your favorite observable analysis engine which corrects bugs introduced by version 1.1.1 and adds a few enhancements. As a reminder, TheHive, our Security Incident Response Platform, can interact with one or several Cortex instances. Moreover, starting from version 1.1.1, Cortex has a two-way integration with MISP.
We highly advise you to upgrade your Cortex in to instance to 1.1.2.
Cortex 1.1.2 – Job Report Example with CERT-SG’s Abuse Finder
Fixed Issues
#27: fixed the daunting error 500 that many users of TheHive encountered when a job is submitted to Cortex.
#29: the MISP expansion modules are now disabled by default to avoid another error 500.
#31: the web interface was displaying SNAPSHOT (oops!) for the Cortex version. It now displays the correct version.
Enhancements
#28: when you enable the MISP expansion modules, Cortex will not be slowed down and starts without delay.
To update your current Cortex installation, follow the instructions of the installation guide. Before doing so, you may want to save the job reports that were not executed via TheHive. Cortex 1 has no persistence and restarting the service will wipe out any existing reports.
Please note that you can install Cortex using an RPM or DEB package, deploy it using an Ansible script, use Docker, install it from a binary or build it from sources.
Support
Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.