When you need to interact with TheHive’s REST API and you ain’t shy of working with Python, TheHive4py is the way to go. It’s a free, open source library we provide to allow you to easily create alert feeders, automate certain tasks like creating cases, assign them to analysts and much more. For example, Synapse, DigitalShadows2TH and Zerofox2TH leverage the library to send alerts to your favourite SIRP/SOAR.
Sometime ago, we decided that it was time to overhaul the whole library and we began working on version 2.0.0 which will be easier to use. It should also support the full set of TheHive’s REST API calls. In the meantime we decided to release version 1.5.0, shortly followed by version 1.5.1 to support some new functionality contributed by our user community and correct a few issues.
New Features Introduced in 1.5.0
#79: Add search support for tasks, contributed by neok0
We released Cortex 2.1.0 as a release candidate back in July 31, 2018 along with TheHive 3.1.0-RC1. By then, the power duo which makes digital forensics, incident response and, to an extent, cyber threat intelligence, better, faster, happier, regular exercising gained the ability to perform active response.
We ate our own dog food for a couple of months. We found bugs. We added enhancements and we listened to the early adopters of these new major versions. And today we are thrilled to announce the availability of the stable release of Cortex 2.1.0 along with TheHive 3.1.0.
Cortex 2.1.0 restores the ability to query the analysis and response engine from MISP for enrichment purposes. A new version of the de facto standard for threat sharing should be released shortly as there are also some API-related issues on its side to make the integration fully working again.
Cortex 2.1.0 also gives you the ability to see the PAP (Permissible Actions Protocol) values for each analyzer as well as any custom cache values you might have configured.
When we’re not busy cooking new features, we go back to the trenches and face incidents like many of our fellow analysts who read our publications and use our tools. To do so, we swap our chef toques for firefighter helmets, not only because such shiny headwear is cool, but mainly because incident response (IR) is, at its very heart, firefighting (minus all the dangerous stuff).
If you think about it, when handling incidents you can see everything from cats in trees (spam) to major fire (APT). Thankfully, there are more cats to bring down than fire to extinguish. That being said, a big herd of cats could be a serious threat to your organization, to your mental health or both.
We tend to forget that incident handlers are humans, not robots. Unlike our metal cyberfriends, we need diversity. We can’t risk insanity like Charlie Chaplin in Modern Times if we can avoid it. Unfortunately IR can be highly repetitive, especially if you only have cats to deal with.
Some could say ‘Nah, this is minor, nothing critical here’ but at some point, an analyst brainwashed by the same tasks again and again will be led to fault. In the worst case scenario, one could see an alert and immediately categorize it as false positive without any further consideration. Because ‘this alert is always a false positive’, until the day it is not…
Automation, a Solution?
Intuitively, we look in the direction of automation in order to minimize what we call ‘zombie’ tasks: highly repetitive and brainless tasks that need to be done. We believe that doing so will allow incident handlers to focus on the analysis and not on the tedious side of IR. Ultimately, we hope it will keep analysts stimulated and in a state of alert. Also, it should reduce time and effort spent on the low-hanging fruits.
One of the most dreary tasks in our opinion is to record the context around an incident.
What is the problem? When did it happen? What’s the origin? Who are the victims? How many are there? Answers to these questions let you have an overview of what is happening and are valuable to correlate incidents. So it is worth taking some minutes to add this information to your case. Sadly, most of the time it will look like a succession of ‘Ctrl+C; Alt+Tab; Ctrl+V’ from your incident source to TheHive. Exactly the kind of tasks we want to forego.
Having identified the threat that apathetic analysts pose, the root cause (highly repetitive tasks) and a solution (automate the recording of incident context), the question of the implementation has been raised.
The first challenge to solve is the number of incident sources. Almost everything can trigger an incident: a firewall, an IDS, antivirus, SIEM, users, etc… So the application must be designed to accept several sources and must permit to easily integrate new ones. And instead of having to configure multiple alert feeders to supply alerts to TheHive, we would have only one. To some extent, it can be assimilated to a meta feeder.
And if the application works as intended, we still have a second challenge. Let’s say you, dear reader, and ourselves use the galaxy renowned Stargazer IDS. Maybe you’d like to include the full packet capture in the case but we wouldn’t. Using the same product doesn’t mean using it the same way. So we have a variety of sources and for each source, we have a variety of configurations and workflows. Hence any app we design needs to accept multiple configurations and workflows for any given source.
Finally: the third challenge. We want to make the most out of TheHive. Creating cases, creating alerts, assigning cases, adding logs, adding observables … all those actions are not an option.
After several trials and failures, we came up with Synapse. Basically it is a Python 3app which sits between TheHive and your incident sources:
To solve the first and third challenge, we rely on connectors. A connector is a Python object dedicated to interact with a security device. In the picture above, you can see the Exchange Connector and TheHive Connector. To extend the number of sources, you just have to develop the connector that corresponds to your device.
Regarding the second challenge, we rely on workflows. Workflows are python scripts who use connectors to automate repetitive tasks when tracking a case. Not happy with the current workflow? Develop your own using the connectors.
At this point, you probably wonder why there’s an API in the picture above. Well, the API is the link between the user and the workflows. By hitting a specific endpoint of the Synapse’s API, the corresponding workflow will be launched. That way the user can choose what to launch, especially if they are only interested in a particular workflow. Moreover, using an API allows us to listen to TheHive’s real-time stream and initiate some actions like closing a QRadar offense when the related case is solved.
At the moment, Synapse includes the Exchange connector and the associated Ews2Case workflow. The workflow features:
Case creation from emails
Adding email bodies to task logs
Adding email replies to the case
Adding email attachments as observables
And of course, everything is done to minimize the number of clicks! Check the workflow documentation to understand how it works under the hood.
We’re still working on the QRadar connector and the associated workflows but if you can’t wait, have a look at the work done by the community like pierrebarlet’s script.
Check it Out
As usual, Synapse is an open source and free software released under the AGPL (Affero General Public License).
hashdd.com is a search engine for file hashes which automatically queries 3rd party services like VirusTotal and enriches the information provided based on the 3rd party data. The analyzer includes two flavors: Status and Detail. The first one is used to query hashdd without an API key for the threat level only. The latter produces additional meta information about the sample, but requires an API key.
Results are displayed in TheHive in the following manner:
URLhaus, a service that shares the latest malware download URLs and reports those sites to their respective hosting companies, can now be queried for domains, URLs and hashes. If the observable is found, available information will be displayed as follows:
Domaintools Risk and Reputation
New Risk and Reputation services from Domaintools have been added as new flavors to the existing Domaintools analyzer set.
The DomainTools Risk Score predicts the risk level and likely threats from a domain that has not been observed in malicious activities, by analyzing various properties of the domain as soon as it is registered.
The Domaintools Reputation Score gives indications about how closely a domain is related to known bad domains, actors, and IPs.
An All New FileInfo
FileInfo performs local static analysis of file observables. It has been completely rewritten from the ground up to be more flexible thus it can easily be enriched with new supported file types and analysis modules. We took this opportunity to merge MsgParser, in charge of extracting and displaying Outlook emails into FileInfo.
As of this release, FileInfo now supports PDF, PE, MS Office documents and Outlook .msg files. We also added support for DDE detection and link extraction in MS Office documents, thanks to Decalage who added this in Oletools since v0.52.
#286 : we updated the way MISP analyzer validates its SSL configuration
Cortex, a free, open source software allows security analysts and threat hunters to analyze and enrich observables (IP addresses, hashes, domains, …) collected in the course of an investigation or received from third parties, for example through MISP, the de facto standard for threat sharing.
On March 29, 2018, we released Cortex 2, a major improvement over the previous version which brought, among other cool features, authentication, caching, multi-tenancy (RBAC) and rate limiting. Instead of deploying several Cortex 1 instances behind reverse proxies which would implement authentification, administrators can deploy a single Cortex 2, create multiple organizations and serve the needs of various information security populations while enjoying extra features.
On May 31, 2018, we published a brand new API guide so that developers can take advantage of the powerful REST API of the product. Sadly, Cortex4py, the FOSS Python library we provide to interact with the API was not compatible with Cortex 2. Until today.
Thanks to the hard work of our dear Nabil Adouani, we are happy to announce the immediate availability of Cortex4py 2.0.0, a complete rewrite of the library in Python 3. Cortex4py 2.0.0 is fully compatible with Cortex 2. However, it doesn’t work with Cortex 1.
While TheHive, the highly popular free and open source Security Incident Response Platform (SIRP) we develop has native support for many Cortex 2 instances, Python developers can leverage Cortex4py to interact with Cortex 2, manage organizations, users, analyzer configurations and analyze observables at scale from alternative SIRPs, SIEMs or custom scripts thanks to the 83 analyzers Cortex 2 has as of June 18, 2018.
To install Cortex4py, use PIP3:
$ sudo -H pip3 install cortex4py
If you are using Python on a Windows operating system, please forgo the sudo command.
Cortex4py 2 comes with a usage guide which includes many examples. For example, if you want to fetch the last 10 successful jobs that have been executed against domain names and display the result summaries of those 10 jobs you could write something like:
Migrating from Cortex4py 1
If you have already written scripts using Cortex4py 1.x (for Cortex 1), we tried to keep the already available methods. However, we recommend you adapt your code to leverage the new Cortex4py 2 classes and methods as soon as feasible. Moreover, the existing scripts must be updated to support authentication if you intend to use them with Cortex 2. Please read the Cortex4py 2 usage guide for more information.
Cortex 2.0.0 is brand new software. As such, it might contain bugs and limitations. If you find any or encounter problems, please ask on our user forum, contact us on Gitter, or send us an email at firstname.lastname@example.org. We are here to help.
After announcing Cortex 2.0.0 and TheHive 3.0.7, the first version of your favorite SIRP that is (supposedly) compatible with the brand-new version of Cortex, last week, we thought it was time to relax and enjoy the upcoming, long Easter weekend, the sunny sky of Paris (if you can pierce the veil of the Forever Grey Cloud™ that is hanging over the city of lights), and great jazz music. Heck, I even tweeted about it … only to be proven wrong by Life (and Murphy).
We literally field tested Cortex 2 for 3 weeks, we squashed bugs here and there, until almost the very last minute before the release. And yet, our QA needs to be improved by leaps and bounds as we had to release Cortex 2.0.1 one day after unveiling 2.0.0 to correct some additional bugs. And then some members of the core team and of our growing user community took it for a spin. And all hell broke lose. Well, almost 🙂
Session collisions (when TheHive and Cortex 2 are used on the same machine), analyzer malfunctions, connectivity problems … issues that were not identified during the testing phase, even in a production environment, where everything worked as expected. And we call this ‘Computer Science’. Right, right…
So we worked hard, took out our Code Hammer (it’s like Thor’s but cyber) and blasted away all the bugs that we found out or that were reported to us (arigato gozaimasu!) and we are happy to announce the immediate availability of Cortex 2.0.2, TheHive 3.0.8, Cortexutils 1.2.3 and Cortex-Analyzers 1.9.2.
TL;DR Install or upgrade Cortex 2.0.2, update Cortexutils, git pull the Cortex-analyzers repo to get the latest version of the repository, upgrade to TheHive 3.0.8, follow the Quick Start Guide and have a drink.
If you have time (which is admittedly quite scarce nowadays), please read on the changelogs:
As stated in the previous post, we will release a new version of Cortex4py in order to make it compatible with Cortex 2, continue the work we started with our MISP Project friends to support MISP attribute enrichment through Cortex 2 (MISP currently only supports enrichment using Cortex 1), and perform a long-overdue overhaul of our documentation. We will also release a brand new version of TheHive4py.
Last but not least, we’ll take a hard look at ourselves and our QA. You expect us from us high quality and we hold ourselves to high standards. And we will deliver.
Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at email@example.com. We are here to help.
Two months ago, TheHive Chefs announced that Cortex passed the 30 analyzers mark as they added HybridAnalysis, EmergingThreats and Shodan, all three contributed by our continuously growing user community.
It’s 2018 already and to wish you a very happy new DFIR year, Nils and Jérôme got out of their way and reviewed many outstanding pull requests for new analyzers and fixed several bugs. Kudos bees!
The latest release of Cortex-Analyzers, v 1.8.0, contains not one, not two, not even three but ten new analyzers! Isn’t that good omen for a fresh new year fighting cybercrime?
The ten new analyzers, described below, are:
Bluecoat: contributed by our longtime friends from CERT La Poste.
MISP WarningLists: Nils strikes again (watch out Jérôme! the youngster is gonna leave you way behind ;).
Onyphe: contributed by Pierre Baudry and Adrien Barchapt. It comes in five different flavors.
PayloadSecurity: submitted by Emmanuel Torquato. The analyzer comes in two flavors.
Robtex: added by… Nils again! It has three flavors.
SinkDB: guess who developed that one? Wow, impressive! How did you figure it out? Yes, Nils!
Tor Blutmagie: contributed by Marc-André Doll.
Tor Project: also contributed by Marc-André Doll.
We would like to wholeheartedly thank all the individuals and teams listed above for their invaluable contributions. So a big merci for your work!
The Bluecoat analyzer queries the Symantec – previously known as Bluecoat – WebPulse site review API for the currently assigned site category of URLs or domains. The analyzer needs no further configuration. When executed through TheHive, the analyzer produces short and long reports as shown below:
The C1fApp analyzer queries the C1fApp service, an Open Source threat feed aggregation application, using the API for IP addresses, domains and URL.
Before using the analyzer, you need to create an account on the C1fApp website and get the associated API key which you’ll need to provide as a value for the key parameter of the analyzer config section of /etc/cortex/application.conf as shown below. Once you’ve done so, you’ll need to restart Cortex.
key="<insert API key here>"
When launched using TheHive, the analyzer produces short and long reports such as the following:
Censys.io continually monitors every reachable server and device on the Internet, so you can search for them and analyze them in real time. Using the corresponding analyzer, information about a website certificate can be obtained using the associated IP, domain or certificate hash.
In order to use this analyzer, an account at censys.io has to be registered and the API ID and secret need to be added to the Cortex configuration file:
uid="<Your ID here>"
key="<Your secret here>"
Once done, you’ll have to restart Cortex. When ran from TheHive, the analyzer produces short and long reports such as the following:
Details about the ports can be obtained with a click on the specific button.
In order to detect false positives soon enough in the analysis process, our good friends at the MISP Project published their so called warning lists which contain lists of well-known services or indicators.
This analyzer queries observables against the MISP warning lists. Observables can be an IP address, a hash, a domain, a FQDN or a URL.
To iterate through all the warning lists, the repository itself must be available on the Cortex instance:
We highly recommend you create a cron entry or use a similar mechanism to keep the lists fresh. While the default path for the lists is the misp-warninglists subdirectory it can be adjusted in the configuration file:
When called from TheHive, the analyzer produces short and long reports as shown below:
As you can see, The MISP WarningLists analyzer checks if the repository is up-to-date 😉
The Onyphe analyzer leverages Onyphe’s API to query the service, which provides data about the IP address space and the publicly available information in a single, handy location.
The service comes in five flavors:
Onyphe_Forward: retrieves forward DNS lookup information we have for the given IPv4/IPv6 address with history of changes.
Onyphe_Geolocate: retrieves geolocation information for the given IPv4/IPv6 address.
Onyphe_Ports: retrieves synscan information we have for the given IPv4/IPv6 address with history of changes.
Onyphe_Reverse: retrieves reverse DNS lookup information we have for the given IPv4/IPv6 address with history of changes.
Onyphe_Threats: retrieves Onyphe threats information on anIPv4/IPv6 address with associated history.
To use the analyzer, you need to create an account on the Onyphe website. Provide the API key associated with your account as a value for the key parameter and add the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.
key = "<insert API key here>"
When ran from TheHive, the analyzer produces short and long reports such as the following:
The PayloadSecurity analyzer let you submit observables to a on-premises PayloadSecurity instance. To use it, you need to create an account on the PayloadSecurity service. Provide the API/secret pair as values for the key and secretparameters, collect the URL and environmentid of the service, and add the lines below to the config section of /etc/cortex/application.conf. Then restart the cortex service.
When launched through TheHive, the analyzer produces short and long reports such as the following:
When collecting data about IPs, domains and FQDNs, Robtex can be a good source of information. According to their statistics, they logged over 20 billion DNS resource records. The corresponding analyzer comes in three flavors:
Robtex_Forward_PDNS_Query: checks domains/FQDNs using the Robtex Passive DNS API
Robtex_IP_Query: checks IPs using the Robtex IP API
Robtex_Reverse_PDNS_Query: checks IPs using the Robtex reverse Passive DNS API
The analyzer uses the free Robtex API which needs no subsequent configuration. However, the free API limits the rate and amount of returned data.
When executed using TheHive, the analyzer produces short and long reports such as the following:
SinkDB is a private service provided by abuse.ch which collects sinkholed IPs. Access to the service is allowed to trusted partners only. If you think you qualify, you can request an access using the form available on the SinkDB website. This is most likely only granted to certain CSIRTs and CERTs and not to individuals.
Provide the API key associated with your account as a value for the key parameter and add the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.
key="<insert API key here>"
When ran from TheHive, the analyzer produces short and long reports such as the following:
Tor Blutmagie analyzer extracts data from torstatus.blutmagie.de and checks if an observable is linked to a Tor node. The observable can be an IP address, a FQDN or a domain.
In order to check if an IP, domain or FQDN is a Tor exit node, this analyzer queries the Tor status service at Blutmagie.de. The analyzer uses a caching mechanism in order to save some time when doing multiple queries, so the configuration includes parameters for the cache directory and the caching duration.
Provide the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.
When ran from TheHive, the analyzer produces short and long reports such as the following:
Tor Project analyzer has also been contributed by Marc-André Doll. As the above analyzer, this one checks if an observable is a Tor exit node. This time, however, the source of information is the official Tor network status which can be queried for IP addresses only.
The accepts another parameter, ttl, which is the threshold in seconds for exit nodes before they get discarded. Provide the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.