Cortex-Analyzers 2.8.0: to infinity and beyond!

Thanks to the community and all the contributors, this release comes with 1 new Analyzer, 2 new Responders, lots of improvements and bug fixes.

But there is more news from the front.

Starting from this milestone, bugfixes and new Analyzers or Responders should be released in a smoother way as we are improving few processes. Some changes and recommandations should appear in the next days for submission, and our release process will be improved to fix bugs easier and release new code faster.

We also plan to offer a better documentation. We already started to publish information regarding each Analyzer and Responder. This is a work in progress, and it will be updated with the current requirements guide.

DomainToolsIris documentation page

For each Analyzer and Responder, a page details the purpose of each flavors, the configuration required and even some screenshots from report samples. It will also allow developers to share their own notes if wanted.

New Analyzers

New Responders

Improvements

  • Refactor Onyphe using new v2 api (#736)
  • Improvement in Shodan: add vulns in template and taxonomies (#772 & #776)
  • Improvement in Mailer responder: tasks support and auth (#764, #737, #379)
  • Improvement in SinkDb: support for new api with new dataTypes supported (#483, #498, #756)

Analyzers

LastInfoSec

LastInfoSec offers innovative and automated solutions to collect data, refine it and turn it into useful and actionable information, quickly available to improve the protection, detection and investigation capabilities of companies and government organizations.

TheHive displays the analyzer results as follows:

Short template for LastInfoSec Report
Long Template for LastInfoSec Report

Onyphe

An important work has been made on Onyphe Analyzer to support APIv2. All 7 flavors from older version have been removed and merged into only one flavor named “Onyphe_Summary”. An API key is still needed to query Onyphe API.

TheHive displays the analyzer results as follows:

Onyphe_Summary short report
Onyphe_Summary long report

Responders

Sendgrid

Sendgrid is a customer communication platform for transactional and marketing email used when you have to ensure that your notifications and transactional emails are delivered quickly and securely.

This analyzer works like the Mailer one, but relying on SendGrid external service to delivery emails.

In order to use the service please follow the instruction being careful to the verify your email address.

VirusTotalDownloader

This responders runs on Observables of type “hash” and allows analyst to download corresponding file from VirusTotal. Once downloaded, the file is added to the current case observables in TheHive.

In order to use this responder, a Premium API key from VirusTotal is needed. An API key from TheHive is also needed to upload the file in the observables list.

Use the responder on the hash to add the sample in your Observables

Fixes and Improvements

  • Fix: some analyzer uses invalid “email” dataType (#799)
  • Fix in MalwareBazaar: wrong dataTypes in config (#794)
  • Fix in PhishTank: the JSON object must be str, not ‘bytes’ (#786)
  • Fix in VMRay: fix error in parsing and workflow (#785 & #784)
  • Fix in Wazuh: ipaddress import missing (#778)
  • Fix in Minemeld Responder: requests missing in requirements (#774)
  • Fix in WOT: moving to new endpoint (#771)
  • DomainTools Iris: update api urls (#760)
  • Fix in ThreatResponse: module_type key removed from response (#759)
  • Fix in Abuse_Finder: pythonwhois dependency (#742)

Get It While Supply Lasts!

If you are still using the old-style way of installing analyzers and responders, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done
for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates` menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Ali Cortex and the 40 Analyzers

Two months ago, TheHive Chefs announced that Cortex passed the 30 analyzers mark as they added HybridAnalysis, EmergingThreats and Shodan, all three contributed by our continuously growing user community.

It’s 2018 already and to wish you a very happy new DFIR year, Nils and Jérôme got out of their way and reviewed many outstanding pull requests for new analyzers and fixed several bugs. Kudos bees!

Snapseed
© Saâd Kadhi

The latest release of Cortex-Analyzers, v 1.8.0, contains not one, not two, not even three but ten new analyzers! Isn’t that good omen for a fresh new year fighting cybercrime?

The ten new analyzers, described below, are:

  1. Bluecoat: contributed by our longtime friends from CERT La Poste.
  2. C1fApp: submitted by Dimitris Lambrou.
  3. Censys.io: developed by Nils Kuhnert, now a full member of TheHive Project, on behalf of CERT-Bund.
  4. MISP WarningLists: Nils strikes again (watch out Jérôme! the youngster is gonna leave you way behind ;).
  5. Onyphe: contributed by Pierre Baudry and Adrien Barchapt. It comes in five different flavors.
  6. PayloadSecurity: submitted by Emmanuel Torquato. The analyzer comes in two flavors.
  7. Robtex: added by… Nils again! It has three flavors.
  8. SinkDB: guess who developed that one? Wow, impressive! How did you figure it out? Yes, Nils!
  9. Tor Blutmagie: contributed by Marc-André Doll.
  10. Tor Project: also contributed by Marc-André Doll.

We would like to wholeheartedly thank all the individuals and teams listed above for their invaluable contributions. So a big merci for your work!

Bluecoat

The Bluecoat analyzer queries the Symantec – previously known as Bluecoat – WebPulse site review API for the currently assigned site category of URLs or domains. The analyzer needs no further configuration. When executed through TheHive, the analyzer produces short and long reports as shown below:

firefox_2018-01-10_11-02-03

Bluecoat Analyzer
TheHive: Bluecoat 1.0 Analyzer – Short and Long Report Samples

C1fApp

The C1fApp analyzer queries the C1fApp service, an Open Source threat feed aggregation application, using the API for IP addresses, domains and URL.

Before using the analyzer, you need to create an account on the C1fApp website and get the associated API key which you’ll need to provide as a value for the key parameter of the analyzer config section of /etc/cortex/application.conf as shown below. Once you’ve done so, you’ll need to restart Cortex.

 C1fApp {
     service="query"
     key="<insert API key here>"
     url="https://www.c1fapp.com/cifapp/api/"
 }

When launched using TheHive, the analyzer produces short and long reports such as the following:

sc-short-c1fapp.png

sc-long-c1fapp.png
TheHive: C1fApp 1.0 Analyzer – Short and Long Report Samples

Censys.io

Censys.io continually monitors every reachable server and device on the Internet, so you can search for them and analyze them in real time. Using the corresponding analyzer, information about a website certificate can be obtained using the associated IP, domain or certificate hash.

In order to use this analyzer, an account at censys.io has to be registered and the API ID and secret need to be added to the Cortex configuration file:

Censys {
    uid="<Your ID here>"
    key="<Your secret here>"
}

Once done, you’ll have to restart Cortex. When ran from TheHive, the analyzer produces short and long reports such as the following:

Censys Short

Censys.io Analyzer
TheHive: Censys 1.0 Analyzer – Short and Long Report Samples

Details about the ports can be obtained with a click on the specific button.

MISP WarningLists

In order to detect false positives soon enough in the analysis process, our good friends at the MISP Project published their so called warning lists which contain lists of well-known services or indicators.

This analyzer queries observables against the MISP warning lists. Observables can be an IP address, a hash, a domain, a FQDN or a URL.

To iterate through all the warning lists, the repository itself must be available on the Cortex instance:

git clone https://github.com/MISP/misp-warninglists

We highly recommend you create a cron entry or use a similar mechanism to keep the lists fresh. While the default path for the lists is the misp-warninglists subdirectory it can be adjusted in the configuration file:

 MISPWarningLists {
     path = "/path/to/misp-warninglists/repository" # Default: "misp-warninglists"
 }

When called from TheHive, the analyzer produces short and long reports as shown below:

firefox_2018-01-10_11-01-46

MISP Warninglists Analyzer
TheHive: MISP WarningLists 1.0 Analyzer – Short and Long Report Samples

As you can see, The MISP WarningLists analyzer checks if the repository is up-to-date 😉

Onyphe

The Onyphe analyzer leverages Onyphe’s API to query the service, which provides data about the IP address space and the publicly available information in a single, handy location.

The service comes in five flavors:

  • Onyphe_Forward: retrieves forward DNS lookup information we have for the given IPv4/IPv6 address with history of changes.
  • Onyphe_Geolocate: retrieves geolocation information for the given IPv4/IPv6 address.
  • Onyphe_Ports: retrieves synscan information we have for the given IPv4/IPv6 address with history of changes.
  • Onyphe_Reverse: retrieves reverse DNS lookup information we have for the given IPv4/IPv6 address with history of changes.
  • Onyphe_Threats: retrieves Onyphe threats information on anIPv4/IPv6 address with associated history.

To use the analyzer, you need to create an account on the Onyphe website. Provide the API key associated with your account as a value for the key parameter and add the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.

Onyphe {
    key = "<insert API key here>"
}

When ran from TheHive, the analyzer produces short and long reports such as the following:

sc-short-onyphe.png

sc-long-onyphe.png
TheHive: Onyphe 1.0 Analyzer – Short and Long Report Samples

PayloadSecurity

The PayloadSecurity analyzer let you submit observables to a on-premises PayloadSecurity instance. To use it, you need to create an account on the PayloadSecurity service. Provide the API/secret pair as  values for the key and secretparameters, collect the URL and environmentid of the service,  and add the lines below to the ​​config section of  /etc/cortex/application.conf. Then restart the cortex service.

PayloadSecurity {
    url = "<insert URL here>"
    key="<insert API key here>"
    secret="<insert secret here>"
    environmentid="<insert environmentid here>"
    verifyssl=True
}

When launched through TheHive, the analyzer produces short and long reports such as the following:

sc-short-payloadsecurity.png

sc-long-payloadsecurity.png
TheHive: PayloadSecurity 1.0 Analyzer – Short and Long Report Samples

Robtex

When collecting data about IPs, domains and FQDNs, Robtex can be a good source of information. According to their statistics, they logged over 20 billion DNS resource records. The corresponding analyzer comes in three flavors:

  • Robtex_Forward_PDNS_Query: checks domains/FQDNs using the Robtex Passive DNS API
  • Robtex_IP_Query: checks IPs using the Robtex IP API
  • Robtex_Reverse_PDNS_Query: checks IPs using the Robtex reverse Passive DNS API

The analyzer uses the free Robtex API which needs no subsequent configuration. However, the free API limits the rate and amount of returned data.

When executed using TheHive, the analyzer produces short and long reports such as the following:

Robtex Short

Robtex Analyzer
TheHive: Robtex 1.0 Analyzer – Short and Long Report Samples

SinkDB

SinkDB is a private service provided by abuse.ch which collects sinkholed IPs. Access to the service is allowed to trusted partners only. If you think you qualify, you can request an access using the form available on the SinkDB website. This is most likely only granted to certain CSIRTs and CERTs and not to individuals.

Provide the API key associated with your account as a value for the key parameter and add the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.

SinkDB {
    key="<insert API key here>"
}

When ran from TheHive, the analyzer produces short and long reports such as the following:

SinkDB Short True

SinkDB Long
TheHive: SinkDB 1.0 Analyzer Short and Long Report Samples

Tor Blutmagie

Tor Blutmagie analyzer extracts data from torstatus.blutmagie.de  and checks if an observable is linked to a Tor node. The observable can be an IP address, a FQDN or a domain.

In order to check if an IP, domain or FQDN is a Tor exit node, this analyzer queries the Tor status service at Blutmagie.de. The analyzer uses a caching mechanism in order to save some time when doing multiple queries, so the configuration includes parameters for the cache directory and the caching duration.

Provide the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.

TorBlutmagie {
    cache {
        duration=3600
        root=/tmp/cortex/tor_project
    }
}

When ran from TheHive, the analyzer produces short and long reports such as the following:

firefox_2018-01-10_11-01-55

Tor Blutmagie Analyzer

Tor Blutmagie Analyzer (2)
TheHive: Tor Blutmagie 1.0 Analyzer – Short and Long Report Samples

Tor Project

Tor Project analyzer has also been contributed by Marc-André Doll. As the above analyzer, this one checks if an observable is a Tor exit node. This time, however, the source of information is the official Tor network status which can be queried for IP addresses only.

The accepts another parameter, ttl, which is the threshold in seconds for exit nodes before they get discarded. Provide the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.

TorProject {
    cache {
        duration=3600
        root=/tmp/cortex/tor_project
        ttl=86400
    }
}

When ran from TheHive, the analyzer produces short and long reports such as the following:

TorProject Short

Tor Project Analyzer
TheHive: Tor Project 1.0 Analyzer – Short and Long Report Samples

Additional Fixes and Improvements

  • #141: Joe Sandbox analyzer now supports API version 2
  • #158: Fix mode when creating FireHOL ipset directory
  • #162: Fix Snort alerts in Cuckoo analyzer
  • #149: Fix the VirusShare hash downloader

Please note that when we fixed the bug in the shell script of VirusShare analyzer, the original Python script was removed.

Update TheHive Report Templates

If you are using TheHive, get the last version of  the report templates and import them into TheHive.

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Correction: January 12, 2018
The post was updated to add the full name of the author of the PayloadSecurity analyzer.