Cortex4py 2 is Out!

Cortex, a free, open source software allows security analysts and threat hunters to analyze and enrich observables (IP addresses, hashes, domains, …) collected in the course of an investigation or received from third parties, for example through MISP, the de facto standard for threat sharing.

On March 29, 2018, we released Cortex 2, a major improvement over the previous version which brought, among other cool features, authentication, caching, multi-tenancy (RBAC) and rate limiting. Instead of deploying several Cortex 1 instances behind reverse proxies which would implement authentification, administrators can deploy a single Cortex 2, create multiple organizations and serve the needs of various information security populations while enjoying extra features.

On May 31, 2018, we published a brand new API guide so that developers can take advantage of the powerful REST API of the product. Sadly, Cortex4py, the FOSS Python library we provide to interact with the API was not compatible with Cortex 2. Until today.

Thanks to the hard work of our dear Nabil Adouani, we are happy to announce the immediate availability of Cortex4py 2.0.0, a complete rewrite of the library in Python 3. Cortex4py 2.0.0 is fully compatible with Cortex 2. However, it doesn’t work with Cortex 1.

While TheHive, the highly popular free and open source Security Incident Response Platform (SIRP) we develop has native support for many Cortex 2 instances, Python developers can leverage Cortex4py to interact with Cortex 2, manage organizations, users, analyzer configurations and analyze observables at scale from alternative SIRPs, SIEMs or custom scripts thanks to the 83 analyzers Cortex 2 has as of June 18, 2018.

Screen Shot 2018-06-18 at 20.01.27.png
Cortex 2: there is more than one way to interact with it

Use It

To install Cortex4py, use PIP3:

$ sudo -H pip3 install cortex4py

If you are using Python on a Windows operating system, please forgo the sudo command.

Usage

Cortex4py 2 comes with a usage guide which includes many examples. For example, if you want to fetch the last 10 successful jobs that have been executed against domain names and display the result summaries of those 10 jobs you could write something like:

Screen Shot 2018-06-18 at 19.58.45.png
Sample Python3 code to retrieve Cortex analyzer results

Migrating from Cortex4py 1

If you have already written scripts using Cortex4py 1.x (for Cortex 1), we tried to keep the already available methods. However, we recommend you adapt your code to leverage the new Cortex4py 2 classes and methods as soon as feasible. Moreover, the existing scripts must be updated to support authentication if you intend to use them with Cortex 2. Please read the Cortex4py 2 usage guide for more information.

Support

Cortex 2.0.0 is brand new software. As such, it might contain bugs and limitations. If you find any or encounter problems, please ask on our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

TheHive 3.0.10 Released

Rejoice DFIR fighters! Your favorite Security Incident Response Platform has just gotten better. TheHive 3.0.10 (codename Cerana 0.10) is just out of the oven. Thomas and Nabil, as the true code Chefs they are, have fixed a number of issues and added a few enhancements in this release. Read on, update and defend your constituency!

Fixes

  • #446: keep observable reports when performing a case merge operation
  • #592: fix the error message when an observable is created with invalid data
  • #583: cases can be merged by using their case ID
  • #584: correct error messages when performing invalid searches
  • #586: a regression was introduced that prevented analyzer names from being reflected in the modal view of mini-reports
  • #512: there were situations were mini-reports won’t display upon job success
  • #522: under certain circumstances, TheHive failed to synchronize MISP events correctly
  • #555: private dashboards must not be searchable
  • #557: fix an error when a merged case is deleted

Enhancements

  • #579: rotate application.log to keep it from becoming huge
  • #564: when submitting a job to Cortex, send the case ID as well
  • #563: poll the status of connectors (MISP, Cortex) every minute and update the UX display accordingly
  • #548: sort related cases by the amount of shared observables
  • #546: display start and end times for closed tasks

Support

Something does not work as expected? You have troubles installing or upgrading? You have spotted new bugs? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

Cortex 2.0.3 Released and Analyzer Updates

There’s a new version of your ultimate observable analysis engine in town : Cortex 2.0.3 is out!

Cortex 2.0.3 contains a few important enhancements over its predecessor and fixes a number bugs as described in the full changelog summarised below. So get it while it’s still hot out of the digital oven and let us know how tasty it is.

analyzeallthethings
Source : Quickmeme.com

Implemented Enhancements

  • #81: reflect proxy changes in the global configuration at the analyzer level
  • #82: display invalid analyzers and let orgadmins delete them
  • #85: allow orgadmins to override the default global report cache.job period per analyzer through the Web UI
  • #86: allow a job to run with arbitrary parameters

Fixed Bugs

  • #75: a version upgrade of an analyzer makes all analyzers invisible in TheHive
  • #80: fix the analyzer configuration dialog to allow orgadmins to override the auto artifact extraction at the analyzer level
  • #83: hit Nabil on the head pretty hard until the analyzer refresh UI button works (well now it does so you can stop hitting poor Nabil’s head).

Analyzer Updates

We took the opportunity of a new release to make a few updates to the public analyzers. Cortex-Analyzers 1.9.3 contains the following changes:

  • Remove the Bluecoat analyzer to comply with the new ‘no scrapping’ ToS imposed by Symantec
  • Fix the default configuration of the Cymon Check IP analyzer
  • Fix the View all VT long template
  • Make the MISP Warning Lists Analyzer ignore case sensitivity when searching for hashes
  • Restrict the Abuse Finder and FileInfo analyzer dependencies to Python 2.7

You can read the full changelog if you like but if you want to enjoy the goods right away, git pull is your friend.

Support

Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

 

Cerana 0.4: Hotfix & Way Beyond

Update: Cerana 0.5 (TheHive 3.0.5) was released on Feb 8, 2018 to fix a regression introduced by Cerana 0.4, pertaining to the observable mini-reports. Please install 3.0.5 instead.

Waiting for the menu, one of the numerous guests of TheHive Project’s world-renowned code restaurant grows impatient. He stands up and walks toward the kitchen, looking for a waiter to chastise. It’s certainly a free meal and, looking at the reviews on CodeAdvisor, a delicious one. But guests shouldn’t wait more than two minutes to get a copy of the menu, right?

As he approaches the door leading to the kitchen, it slams open and an all-smiling, all-French-looking man dressed in a bee costume comes out carrying hot plates. Peering at their content, the once angry guest cheers up as he stares down at Cerana 0.4 or TheHive 3.0.4 if you prefer. If the looks and smells of the dish match its taste, this will be a terrific meal.

As he sees how Cerana 0.4 prompted a change of heart of the guest, the waiter coming out of the kitchen where TheHive Chefs were busy creating this new recipe goes on describing its content.

Mon bon Monsieur‘, he starts in near-perfect Parisian-English accent, ‘this is the plat du jour, or the dish of the day if you prefer. Even though the mighty Chefs insist on calling it a hotfix, not only it fixes eleven bugs in previous versions but it adds four new features and two enhancements. Would you like me to describe them?’

dt010205.gif
Source : dilbert.com © Scott Adams

The guest, feeling hungry, his mouth watery, but feeling bad about what he was about to say to the waiter just a few seconds ago nods. So the waiter goes on and describe how the latest release of TheHive will serve his hunger for efficient though free security incident response platforms even better.

Fixed Bugs

Enhancements

New Features

At this point, the guest cannot control his lust so he snatches one of the plates out the waiter’s hands, runs back to his table and start installing Cerana 0.4 to unleash its power and use it to fight cybercrime. Bon appétit !

Feeling Generous? Donate!

As you know, we are a FOSS project and donations are always welcome to make our products even better for the community.

All donations go to Creative Source, the non-profit organization we have created, and we will use them to improve TheHive, Cortex & Hippocampe but also to develop (even better) integrations with other FOSS solutions such as MISP.

So if you are feeling generous, please contact us at support@thehive-project.org.

Creative Source can also provide so-called professional, entreprise-grade support, help integrating the products, train your analysts before they drain or assist you in specific areas such as developing in-house analyzers for Cortex.

Download & Get Down to Work

If you have an existing installation of TheHive, please follow the migration guide.

If you are performing a fresh installation, read the installation guide corresponding to your needs and enjoy. Please note that you can install TheHive using an RPM or DEB package, use Docker, install it from a binary or build it from sources.

Support

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

CorrectionFeb 6, 2018
An earlier version of this post contained an incomplete sentence.

Ali Cortex and the 40 Analyzers

Two months ago, TheHive Chefs announced that Cortex passed the 30 analyzers mark as they added HybridAnalysis, EmergingThreats and Shodan, all three contributed by our continuously growing user community.

It’s 2018 already and to wish you a very happy new DFIR year, Nils and Jérôme got out of their way and reviewed many outstanding pull requests for new analyzers and fixed several bugs. Kudos bees!

Snapseed
© Saâd Kadhi

The latest release of Cortex-Analyzers, v 1.8.0, contains not one, not two, not even three but ten new analyzers! Isn’t that good omen for a fresh new year fighting cybercrime?

The ten new analyzers, described below, are:

  1. Bluecoat: contributed by our longtime friends from CERT La Poste.
  2. C1fApp: submitted by Dimitris Lambrou.
  3. Censys.io: developed by Nils Kuhnert, now a full member of TheHive Project, on behalf of CERT-Bund.
  4. MISP WarningLists: Nils strikes again (watch out Jérôme! the youngster is gonna leave you way behind ;).
  5. Onyphe: contributed by Pierre Baudry and Adrien Barchapt. It comes in five different flavors.
  6. PayloadSecurity: submitted by Emmanuel Torquato. The analyzer comes in two flavors.
  7. Robtex: added by… Nils again! It has three flavors.
  8. SinkDB: guess who developed that one? Wow, impressive! How did you figure it out? Yes, Nils!
  9. Tor Blutmagie: contributed by Marc-André Doll.
  10. Tor Project: also contributed by Marc-André Doll.

We would like to wholeheartedly thank all the individuals and teams listed above for their invaluable contributions. So a big merci for your work!

Bluecoat

The Bluecoat analyzer queries the Symantec – previously known as Bluecoat – WebPulse site review API for the currently assigned site category of URLs or domains. The analyzer needs no further configuration. When executed through TheHive, the analyzer produces short and long reports as shown below:

firefox_2018-01-10_11-02-03

Bluecoat Analyzer
TheHive: Bluecoat 1.0 Analyzer – Short and Long Report Samples

C1fApp

The C1fApp analyzer queries the C1fApp service, an Open Source threat feed aggregation application, using the API for IP addresses, domains and URL.

Before using the analyzer, you need to create an account on the C1fApp website and get the associated API key which you’ll need to provide as a value for the key parameter of the analyzer config section of /etc/cortex/application.conf as shown below. Once you’ve done so, you’ll need to restart Cortex.

 C1fApp {
     service="query"
     key="<insert API key here>"
     url="https://www.c1fapp.com/cifapp/api/"
 }

When launched using TheHive, the analyzer produces short and long reports such as the following:

sc-short-c1fapp.png

sc-long-c1fapp.png
TheHive: C1fApp 1.0 Analyzer – Short and Long Report Samples

Censys.io

Censys.io continually monitors every reachable server and device on the Internet, so you can search for them and analyze them in real time. Using the corresponding analyzer, information about a website certificate can be obtained using the associated IP, domain or certificate hash.

In order to use this analyzer, an account at censys.io has to be registered and the API ID and secret need to be added to the Cortex configuration file:

Censys {
    uid="<Your ID here>"
    key="<Your secret here>"
}

Once done, you’ll have to restart Cortex. When ran from TheHive, the analyzer produces short and long reports such as the following:

Censys Short

Censys.io Analyzer
TheHive: Censys 1.0 Analyzer – Short and Long Report Samples

Details about the ports can be obtained with a click on the specific button.

MISP WarningLists

In order to detect false positives soon enough in the analysis process, our good friends at the MISP Project published their so called warning lists which contain lists of well-known services or indicators.

This analyzer queries observables against the MISP warning lists. Observables can be an IP address, a hash, a domain, a FQDN or a URL.

To iterate through all the warning lists, the repository itself must be available on the Cortex instance:

git clone https://github.com/MISP/misp-warninglists

We highly recommend you create a cron entry or use a similar mechanism to keep the lists fresh. While the default path for the lists is the misp-warninglists subdirectory it can be adjusted in the configuration file:

 MISPWarningLists {
     path = "/path/to/misp-warninglists/repository" # Default: "misp-warninglists"
 }

When called from TheHive, the analyzer produces short and long reports as shown below:

firefox_2018-01-10_11-01-46

MISP Warninglists Analyzer
TheHive: MISP WarningLists 1.0 Analyzer – Short and Long Report Samples

As you can see, The MISP WarningLists analyzer checks if the repository is up-to-date 😉

Onyphe

The Onyphe analyzer leverages Onyphe’s API to query the service, which provides data about the IP address space and the publicly available information in a single, handy location.

The service comes in five flavors:

  • Onyphe_Forward: retrieves forward DNS lookup information we have for the given IPv4/IPv6 address with history of changes.
  • Onyphe_Geolocate: retrieves geolocation information for the given IPv4/IPv6 address.
  • Onyphe_Ports: retrieves synscan information we have for the given IPv4/IPv6 address with history of changes.
  • Onyphe_Reverse: retrieves reverse DNS lookup information we have for the given IPv4/IPv6 address with history of changes.
  • Onyphe_Threats: retrieves Onyphe threats information on anIPv4/IPv6 address with associated history.

To use the analyzer, you need to create an account on the Onyphe website. Provide the API key associated with your account as a value for the key parameter and add the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.

Onyphe {
    key = "<insert API key here>"
}

When ran from TheHive, the analyzer produces short and long reports such as the following:

sc-short-onyphe.png

sc-long-onyphe.png
TheHive: Onyphe 1.0 Analyzer – Short and Long Report Samples

PayloadSecurity

The PayloadSecurity analyzer let you submit observables to a on-premises PayloadSecurity instance. To use it, you need to create an account on the PayloadSecurity service. Provide the API/secret pair as  values for the key and secretparameters, collect the URL and environmentid of the service,  and add the lines below to the ​​config section of  /etc/cortex/application.conf. Then restart the cortex service.

PayloadSecurity {
    url = "<insert URL here>"
    key="<insert API key here>"
    secret="<insert secret here>"
    environmentid="<insert environmentid here>"
    verifyssl=True
}

When launched through TheHive, the analyzer produces short and long reports such as the following:

sc-short-payloadsecurity.png

sc-long-payloadsecurity.png
TheHive: PayloadSecurity 1.0 Analyzer – Short and Long Report Samples

Robtex

When collecting data about IPs, domains and FQDNs, Robtex can be a good source of information. According to their statistics, they logged over 20 billion DNS resource records. The corresponding analyzer comes in three flavors:

  • Robtex_Forward_PDNS_Query: checks domains/FQDNs using the Robtex Passive DNS API
  • Robtex_IP_Query: checks IPs using the Robtex IP API
  • Robtex_Reverse_PDNS_Query: checks IPs using the Robtex reverse Passive DNS API

The analyzer uses the free Robtex API which needs no subsequent configuration. However, the free API limits the rate and amount of returned data.

When executed using TheHive, the analyzer produces short and long reports such as the following:

Robtex Short

Robtex Analyzer
TheHive: Robtex 1.0 Analyzer – Short and Long Report Samples

SinkDB

SinkDB is a private service provided by abuse.ch which collects sinkholed IPs. Access to the service is allowed to trusted partners only. If you think you qualify, you can request an access using the form available on the SinkDB website. This is most likely only granted to certain CSIRTs and CERTs and not to individuals.

Provide the API key associated with your account as a value for the key parameter and add the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.

SinkDB {
    key="<insert API key here>"
}

When ran from TheHive, the analyzer produces short and long reports such as the following:

SinkDB Short True

SinkDB Long
TheHive: SinkDB 1.0 Analyzer Short and Long Report Samples

Tor Blutmagie

Tor Blutmagie analyzer extracts data from torstatus.blutmagie.de  and checks if an observable is linked to a Tor node. The observable can be an IP address, a FQDN or a domain.

In order to check if an IP, domain or FQDN is a Tor exit node, this analyzer queries the Tor status service at Blutmagie.de. The analyzer uses a caching mechanism in order to save some time when doing multiple queries, so the configuration includes parameters for the cache directory and the caching duration.

Provide the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.

TorBlutmagie {
    cache {
        duration=3600
        root=/tmp/cortex/tor_project
    }
}

When ran from TheHive, the analyzer produces short and long reports such as the following:

firefox_2018-01-10_11-01-55

Tor Blutmagie Analyzer

Tor Blutmagie Analyzer (2)
TheHive: Tor Blutmagie 1.0 Analyzer – Short and Long Report Samples

Tor Project

Tor Project analyzer has also been contributed by Marc-André Doll. As the above analyzer, this one checks if an observable is a Tor exit node. This time, however, the source of information is the official Tor network status which can be queried for IP addresses only.

The accepts another parameter, ttl, which is the threshold in seconds for exit nodes before they get discarded. Provide the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.

TorProject {
    cache {
        duration=3600
        root=/tmp/cortex/tor_project
        ttl=86400
    }
}

When ran from TheHive, the analyzer produces short and long reports such as the following:

TorProject Short

Tor Project Analyzer
TheHive: Tor Project 1.0 Analyzer – Short and Long Report Samples

Additional Fixes and Improvements

  • #141: Joe Sandbox analyzer now supports API version 2
  • #158: Fix mode when creating FireHOL ipset directory
  • #162: Fix Snort alerts in Cuckoo analyzer
  • #149: Fix the VirusShare hash downloader

Please note that when we fixed the bug in the shell script of VirusShare analyzer, the original Python script was removed.

Update TheHive Report Templates

If you are using TheHive, get the last version of  the report templates and import them into TheHive.

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Correction: January 12, 2018
The post was updated to add the full name of the author of the PayloadSecurity analyzer.

The Perfect Christmas Gift

George Abitbol* doesn’t feel well. Christmas is approaching at a fast pace and the gift he ordered days ago for his girlfriend didn’t find its way to his mailbox yet. He checked it out three times today and save for some spam catalogues on how to take care of his handsome silhouette, nothing resembling a gift showed up.

IMG_4107.jpg
Picture by Saâd Kadhi

He tried to call the French parcel service to know the whereabouts of the luxurious, limited version of the organic sweet potato chips his lovely Jacqueline* likes so much, which sells for four times the regular price (to bear the cost of the enhanced packaging, certainly), but he couldn’t get hold of a living soul all day long. When he placed the order, the delivery was supposed to be lightning fast. It turned out to be a false promise.

With a sinking heart, he climbs back the stairs leading to his apartment, fetches his laptop and sits on his club chair. With his headphones on, immersed in the wonderful jazz of Christian Scott, he wanders randomly through online shopping sites trying to make out his mind on what other presents he could get for his dear Jacqueline, in time for Christmas.

In the middle of the track called Encryption, featuring the uncanny Elena Pinderhugues on flute, a Twitter notification resonates in his ears. He checks it out and learn that TheHive Chefs, as true and elegant gentlemen, have published a new training VM for Cerana 0.3 (a.k.a. TheHive 3.0.3), including Cortex 1.1.4 and the latest set of Cortex-Analyzers.

George loves bees in all shapes and forms, including digital ones so he swiftly downloads the new VM and as the cautious person he is, he verifies the file’s SHA256 hash: 86a87b70627e8db672c57cb57821461f2564ae9b8087cc22fdd1e7a599c16aedWonderful! Everything checks out beautifully. He then imports the file in his VM software, starts the virtual machine and logs in as thehive then types in thehive1234 when asked for the password.

He thoroughly reads the documentation to configure various analyzers and integrate his favourite Security Incident Response Platform with MISP.  A few minutes later, his VM is ready for prime time and he starts playing with the new multi-source dashboards and interacting with fellow analysts on Gitter.

And he totally forgets about Jacqueline’s gift.

(*) Any resemblance to real and actual names is purely coincidental.

Correction: Dec 23, 2017
An earlier version of this post was referring to a previous training VM that included Cerana 0.2, a version affected by a privilege escalation vulnerability which was corrected in Cerana 0.3. Some typos were corrected as well.

Cerana 0.2: X-Pack Auth, Multi-source Dashboards

TheHive Chefs are happy to announce the immediate availability of their latest recipe: Cerana 0.2 (a.k.a. TheHive 3.0.2).

While TheHive 3.0.0 brought you dynamic dashboards among other niceties, the latest (and, of course, the greatest) version of your Mom’s favourite Security Incident Response Platform fixes a bug spotted by our longtime supporter Megan Roddie (merci !). Indeed, Nabil was running low on coffee so he didn’t make the necessary changes to support the new sighted toggle introduced by Cerana for file observables.

Cerana 0.2 also adds X-Pack authentication for Elasticsearch, a feature contributed by srilumpa. Thanks! To enable this functionality, and assuming the X-Pack plugin for Elasticsearch is installed, add the following section to /etc/thehive/application.conf:

search.username = "jessica"​​​​​

​search.password = "drink-beat-repeat"

Last but not least, we decided to make dynamic dashboards even more powerful. You can now create new graphs that support multiple series from multiple entities (or sources).

multiline2
Multi-line Dashboards Example — Number of IOCs imported from MISP vs. those imported from other sources

As Christmas is approaching, go ahead and play with dynamic dashboards to impress your management as soon as 2018 rears its head or truly drive your CTI and DFIR activities and plan well ahead how you should improve automation or collaboration (or beg for additional headcount).

multiline1.png
Multi-line Dashboards Example — How to create one

Ain’t that nifty? Who said bees aren’t nice? Joyeux Noël !

 

Feeling Generous? Donate!

As you know, we are a FOSS project and donations are always welcome to make our products even better for the community.

All donations go to Creative Source, the non-profit organization we have created, and we will use them to improve TheHive, Cortex & Hippocampe but also to develop (even better) integrations with other FOSS solutions such as MISP.

So if you are feeling generous, contact us at support@thehive-project.org. Of course the funds may also be used to keep Nabil happy by providing a steady flow of caffeine. 😉

Creative Source can also provide so-called professional, entreprise-grade support, help integrating the products, train your analysts before they drain or assist you in specific areas such as developing in-house analyzers for Cortex.

Download & Get Down to Work

If you have an existing installation of TheHive, please follow the migration guide.

If you are performing a fresh installation, read the installation guide corresponding to your needs and enjoy. Please note that you can install TheHive using an RPM or DEB package, use Docker, install it from a binary or build it from sources.

Support

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.