Cortex-Analyzers 2.8.0: to infinity and beyond!

Thanks to the community and all the contributors, this release comes with 1 new Analyzer, 2 new Responders, lots of improvements and bug fixes.

But there is more news from the front.

Starting from this milestone, bugfixes and new Analyzers or Responders should be released in a smoother way as we are improving few processes. Some changes and recommandations should appear in the next days for submission, and our release process will be improved to fix bugs easier and release new code faster.

We also plan to offer a better documentation. We already started to publish information regarding each Analyzer and Responder. This is a work in progress, and it will be updated with the current requirements guide.

DomainToolsIris documentation page

For each Analyzer and Responder, a page details the purpose of each flavors, the configuration required and even some screenshots from report samples. It will also allow developers to share their own notes if wanted.

New Analyzers

New Responders

Improvements

  • Refactor Onyphe using new v2 api (#736)
  • Improvement in Shodan: add vulns in template and taxonomies (#772 & #776)
  • Improvement in Mailer responder: tasks support and auth (#764, #737, #379)
  • Improvement in SinkDb: support for new api with new dataTypes supported (#483, #498, #756)

Analyzers

LastInfoSec

LastInfoSec offers innovative and automated solutions to collect data, refine it and turn it into useful and actionable information, quickly available to improve the protection, detection and investigation capabilities of companies and government organizations.

TheHive displays the analyzer results as follows:

Short template for LastInfoSec Report
Long Template for LastInfoSec Report

Onyphe

An important work has been made on Onyphe Analyzer to support APIv2. All 7 flavors from older version have been removed and merged into only one flavor named “Onyphe_Summary”. An API key is still needed to query Onyphe API.

TheHive displays the analyzer results as follows:

Onyphe_Summary short report
Onyphe_Summary long report

Responders

Sendgrid

Sendgrid is a customer communication platform for transactional and marketing email used when you have to ensure that your notifications and transactional emails are delivered quickly and securely.

This analyzer works like the Mailer one, but relying on SendGrid external service to delivery emails.

In order to use the service please follow the instruction being careful to the verify your email address.

VirusTotalDownloader

This responders runs on Observables of type “hash” and allows analyst to download corresponding file from VirusTotal. Once downloaded, the file is added to the current case observables in TheHive.

In order to use this responder, a Premium API key from VirusTotal is needed. An API key from TheHive is also needed to upload the file in the observables list.

Use the responder on the hash to add the sample in your Observables

Fixes and Improvements

  • Fix: some analyzer uses invalid “email” dataType (#799)
  • Fix in MalwareBazaar: wrong dataTypes in config (#794)
  • Fix in PhishTank: the JSON object must be str, not ‘bytes’ (#786)
  • Fix in VMRay: fix error in parsing and workflow (#785 & #784)
  • Fix in Wazuh: ipaddress import missing (#778)
  • Fix in Minemeld Responder: requests missing in requirements (#774)
  • Fix in WOT: moving to new endpoint (#771)
  • DomainTools Iris: update api urls (#760)
  • Fix in ThreatResponse: module_type key removed from response (#759)
  • Fix in Abuse_Finder: pythonwhois dependency (#742)

Get It While Supply Lasts!

If you are still using the old-style way of installing analyzers and responders, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done
for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates` menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Cortex-Analyzers 2.4.0: 138 Ways to Analyze, 10 Methods to Respond

Guess what? TheHive Project is still alive and well, as Saâd already mentioned in a previous blog post.

We’ve been certainly very busy lately, preparing the upcoming release of TheHive 4 and doing many other things beside working on our FOSS project. As a result, it took us a rather long time to merge several community contributions and reduce the sizeable pile of pull requests.

We would like to thank our contributors for their patience and we hope the cyberdefenders out there will enjoy the brand new Cortex-Analyzers 2.4.0 release, with many new analyzers, responders and some bug fixes & improvements, bringing the total to a whooping 138 analyzers (counting all flavors) and 10 responders!

Additionally, with this release, all analyzers are now using Python 3. No more Python 2 technodebt!

Photo by Saâd Kadhi

What’s New?

New Analyzers

8 new analyzers have been added to this release:

1 analyzer has new flavors:

New Responders

3 new responders have been added:

Overview of the New Analyzers

DomainToolsIris

This analyzer looks up domain names, IP addresses, e-mail addresses, and SSL hashes using the popular DomainTools Iris service API.

The analyzer comes in 2 flavors:

  • DomainToolsIris_Investigate: use DomainTools Iris API to investigate a domain.
  • DomainToolsIris_Pivot: use DomainTools Iris API to pivot on ssl_hash, ip, or email.

A valid DomainTools API integration subscription is needed to run this analyzer.

TheHive displays the analyzer results as follows:

DomainToolsIris short report
DomainToolsIris long report

EmailRep

The EmailRep analyzer checks the reputation of an email address against the emailrep.io database.

IPInfo

This analyzer accesses IP-centric features provided by ipinfo.io. While the EmailRep API can be used without a token for limited usage, the ipinfo.io analyzer requires the configuration of an API token before use.

Maltiverse

This analyzer lets you query the free Maltiverse Threat Intelligence platform for enrichment information about a particular hash, domain, ip or url.

TheHive displays the analyzer results as follows:

Maltiverse short report
Maltiverse long report

MalwareClustering

Andrea Garavaglia contributed this one a long time ago and we finally merged it into the Cortex-Analyzers repository. Andrea gave a talk about the background of this analyzer at the fourth MISP summit. You can watch it here.

In order to use the analyzer, you need to point it to a Neo4j server (you need to supply the host, port, login & password).

PaloAlto Autofocus

This analyzer lets you leverage PaloAlto Autofocus services. Provided you are an Autofocus customer and you have access to their API, you need to configure the analyzer with your username and a token key.

The analyzer comes with 3 flavors:

  • AUTOFOCUS_GetSampleAnalysis lets you request a full report for a given hash.
  • AUTOFOCUS_SearchIOC lets you research for samples linked to specific IoCs with datatypes like domain, fqdn, user-agent, imphash, ip, mutex, tag and url. Please note that mutex and tag are not default datatypes in TheHive. You need to create them in TheHive before you can leverage them.
  • AUTOFOCUS_SearchJSON lets you research for samples based on a complex JSON query.

Important: TheHive has no templates corresponding to this analyzer have been published yet. They will be provided in the near future.

SpamhausDBL

This analyzer performs reputation lookups of a domain or a fqdn against Spamhaus Domain Block List (DBL).

TheHive displays the analyzer results as follows:

SpamhausDBL short report
SpamhausDBL long report

TeamCymruMHR

This analyzer queries Team Cymru’s Malware Hash Registry for known malware hashes (MD5 or SHA-1). If it is malware and known by the service, it returns the last time it has been seen along with an approximate anti-virus detection percentage.

Overview of the New Responders

KnowBe4

This responder allows the integration between TheHive/Cortex and KnowBe4’s User Events API.
If a mail observable is tagged with a specified tag, corresponding to the responder’s configuration (e.g. phished), then the associated user will have a custom event added to their profile in KnowBe4.

A valid account on KnowBe4 and an API key are required to run this responder.

Minemeld

This responder sends observables you select to a Palo Alto Minemeld instance.

To run this responder, a MineMeld Threat Intelligence Sharing account is needed.

Wazuh

This responder performs actions on Wazuh, the open source security monitoring platform. It currently supports ad-hoc firewall blocking of ip observables.

Improvements

New PassiveTotal flavors

Thanks to Brandon Dixon, the PassiveTotal analyzer gains 3 new flavors, bringing the total to 11:

  • PassiveTotal_Trackers let you make tracker lookups on observables of type domain, fqdn and ip.
  • PassiveTotal_Host_Pairs let you make host pair lookups on observables of type domain, fqdn and ip.
  • PassiveTotal_Components lets you make components lookup on observables of type domain, fqdn and ip.

They come with their own report templates.

GreyNoise Analyzer

The analyzer has been updated to support GreyNoise API v2, thanks to the contribution of Whitney Champion (#562).

New Data Types Supported by Some Analyzers

  • VirusTotal_GetReporthas been updated to allow requests for observables of type fqdn.
  • Threatcrowd has been updated to allow requests for observables of type domain.
  • Shodan has been updated to allow requests for observables of type fqdn.

Fixes

  • [#602] The MISP analyzer was bumped to version 2.1 and is ready to use PyMISP 2.4.120.

Get It While Supply Lasts!

I’m Hype

If you are using the dockerized analyzers & responders, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button.

I’m Country

If you are still using the old-style way of installing analyzers and responders, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

More than 120 ways of Analyzing your Observables

There’s a new version of Cortex-Analyzers in cybertown and its has an awesome, mind-blowing name and that’s… wait for it… wait for it: 2.1.0.

In this new release, we added two analyzers which bring the total number to more than 120:

We could not duly test DNSSinkhole since we do not have access to the associated service. So we would really appreciate it if you could test it and let us know whether it works or not.

Others analyzers have been fixed or improved:

DNSSinkhole

This analyzer lets you check if an IP address has been registered in your DNS sinkhole. TheHive displays the analyzer results as follows:

DNSSinkhole analyzer: long report
DNSSinkhole analyzer: short report

TalosReputation

This analyzer lets you determine whether an IP address has been reported as a threat on Cisco Talos Intelligence service. No special access to the service is required to run the analyzer.

TheHive displays the analyzer results as follows:

Talos Reputation: long report
Talos Reputation analyzer : short report

Crt.sh

This analyzer has been enriched to display SHA-1 fingerprints. The long report format has been updated to reflect this new information.

Crt.sh analyzer: long report

FileInfo

FileInfo has been updated and is now able to parse PDF files and extract IOCs such as URLs, hosts, domains, IPs, hashes and many more.The analyzer does also support the last version of the extract-msg library.

FileInfo analyzer: IOC Parser long report
FileInfo analyzer: IOC Parser short report

VirusTotal and Python3

The VirusTotal analyzer, including all its flavours, now uses Python3 and an updated virustotal-api library.

Yeti API key

An optional API key can now be configured and used by the Yeti analyzer.

Malwares_GetReport

A hash computation has been fixed in this analyzer.

EMLParser

A first fix has been introduced to avoid this analyzer to crash when there is no content-description in content_header, and a second has been added to correct a header display issue.

IBM XForce Lookup

The analyzer has been improved to allow users to add a trailing / at the end of the API URL without breaking everything.

Updating your Analyzers in Cortex 2.x

Each analyzer and responder comes with its own, pip compatible requirements.txt file. Run the following commands to update your Cortex analyzers to the latest version:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip2 install -U -r $I || true; done && \
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, do not forget to login to Cortex as an orgadmin and click on the Refresh Analyzers button. Refer to the online Cortex documentation for further details.

Dockerised Analyzers

Cortex 3.x gives you the opportunity to run dockerised analyzers and responders. This means that you no longer have to download all the git repository of Cortex-Analyzers and run lengthy commands to update your analyzers and responders.

If you want to use dockerised analyzers and responders, ensure that the URL of the catalog.json file corresponding to the Cortex-Analyzers repository is registered in application.conf. Please note that this won’t work if you are tracking the stable catalog.

After doing so, do not forget to login to Cortex as an orgadmin, click on the Refresh Analyzers button, then Disable and Enable again each analyzer and responder. Analyzer (and responder) updates should occur automatically as long as docker.autoUpdate is set to true in application.conf (this is the default setting).

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Cortex 101: Dissecting Observables a Hundred Ways

TheHive Project’s Code Chefs are glad to announce that, thanks to the precious contributions of the user community, Cortex has broken the one hundred analyzer mark.

Cortex-Analyzers version 1.14.0 is out and includes new analyzers, some improvements and some bug fixes.

New Analyzers

New and enhanced analyzers, described below, are:

  • Cisco Investigate by Cisco Umbrella Research @opendns
  • Proofpoint Forensics Lookup by Emmanuel Torquato
  • Proofpoint Threat Insight Forensics Analyzer by Emmanuel Torquato
  • RecordedFuture by jojoob
  • urlscan.io search by ninoseki
  • Google DNS over HTTP by 0xswitch
  • RTF files support in FileInfo by Nils Kuhnert
  • Datascan and Inetnum flavors in Onyphe analyzer by Pierre Baudry and Adrien Barchapt

Again, huge thanks for the awesome work that has been performed by all our contributors!

Investigate

Cisco Umbrella Investigate provides threat intelligence about domains and IP addresses accross the Internet. The analyzer can be used to query the Cisco Umbrella (formerly OpenDNS) API and get information about an IP or a domain name. An API key is required to use this analyzer.

Results are displayed in TheHive in the following manner:

sc-investigate-short

sc-investigate-long
Cisco Investigate: short and long reports

Proofpoint Forensics Lookup

According to Proofpoint’s website, the Forensics API allows insight in detailed forensic evidences about individual threats or compaigns. The analyzer can be used to check observables against given indicators of compromise stored in the ProofPoint service.

Unfortunately, there are currently no sample report screenshots available, because TheHive’s Core Team does not have access to Proofpoint services. Also, due to the same reason, this analyzer could not be tested by us. If you have access to the service and can test the analyzer and/or provide report screenshots, please let us know.

RecordedFuture

This analyzer lets you get the latest risk data from RecordedFuture for a hash, domain or an IP address. It can be used to query the API and get information. An API key is required to use this analyzer.

Results are displayed in TheHive in the following manner:

sc-recordedfuture-short

sc-recordedfuture-long
RecordedFuture: short and long reports

Urlscan.io search

Urlscan.io is a service that scans and analyzes websites. Submitted pages will be browsed like a regular user would do and every activity gets recorded. The analyzer submitted by ninoseki queries urlscan without initiating a scan which would be publicly visible on the website. Accepted datatypes for this analyzer are URL, domain, hash and IP.

The templates which display the results of the analyzer look like the following screenshots:

Screenshot from 2018-10-25 09-55-34

Screenshot from 2018-10-25 09-48-58
Urlscan.io: short and long reports

Google DNS over HTTP

This analyzer provides DNS information for an IP, a domain or a FQDN by making calls to Google DNS-over-HTTP (DoH). No API key is required.

Results are displayed in TheHive in the following manner:sc-googleDNS-short

sc-googleDNS-long
Google DNS: short and long reports

RTF files support in FileInfo

The FileInfo meta analyzer has been improved and now leverages the rtfobj tool provided in the Oletools suite by Decalage.

Results are displayed in TheHive in the following manner:

sc-fileinfo_RTF-short

sc-rtfobj-long
FileInfo with rtfobj: short and long reports

Datascan and Inetnum flavors in Onyphe analyzer

The Onyphe analyzer has been enhanced with two new flavors. Datascan provides information about known open ports on a specific IP, and Inetnum enumerates all known network information about the analyzed IP address.

An API key is required to use the analyzer and can be obtained by creating an account on the Onyphe website.

Results are displayed in TheHive in the following manner:

sc-onyphe-inetnum-short

sc-onyphe_inetnum-long
Onyphe  Inetnum: short and long reports

sc-onyphe_datascan-short

sc-onyphe_datascan-long
Onyphe Datascan: long and short reports

Bug fixes and enhancements

  • #248: Improve error msg when VT Get Report does not have an entry for
  • #323: Fix an issue with HybridAnalysis analyzer filenames handler
  • #329: Enhance PassiveTotal Passive DNS report
  • #348: Fix a typo in URLhaus’s long.html
  • #356 : Force the use of Python3 in MISP analyzer
  • #362: Fix file not found issue and empty result set in CERT.at passive DNS analyzer

Get It While Supply Lasts!

Each analyzer comes with its own, pip compatible requirements.txt file. To update your Cortex analyzers to 1.14.0, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip2 install -U -r $I || true; done && \
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, do not forget to login to Cortex as an orgadmin and click on the Refresh Analyzers button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, get the latest version of  the report templates and import them into TheHive.

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Keep Them Coming: Hashdd, URLhaus & a Revamped FileInfo analyzer

We are happy to announce the immediate availability of a new major version of Cortex-Analyzers. Version 1.11.0 includes two brand new analyzers, several updates and a few bug fixes:

  • Hashdd, contributed by  iosonogio
  • URLhaus, contributed by  ninoseki
  • The Domaintools analyzer has been updated with two new flavors: Risk and Reputation
  • The VirusTotal analyzer can (finally!) get reports for URL observables
  • MsgParser and File_Info have been merged in a new, shiny, completely rewritten FileInfo analyzer

As we are approaching the 90 analyzers mark, we wholeheartedly thank our user community for continuously contributing new analyzers, testing them and helping us improve the existing ones.

Important Notice

We made significant changes in this release in the analyzers and short reports. Prior to Cortex-Analyzers 1.11.0, the summary() function in the analyzer code generates a result such as:

"taxonomies": [{
     "level": "suspicious",
     "namespace": "DT",
     "predicate": "Risk",
     "value": "\"10\""
 }]

Double quotes were included in the resulting value. We decided to update the summary() function and make it generate the same result without double quotes :

"taxonomies": [{
     "level": "suspicious",
     "namespace": "DT",
     "predicate": "Risk",
     "value": "10"
 }]

To display the double quotes in TheHive, we updated all short reports to include them:

<span class="label" ng-repeat="t in content.taxonomies" ng-class="{'info': 'label-info', 'safe': 'label-success', 'suspicious': 'label-warning', 'malicious':'label-danger'}[t.level]">
     {{t.namespace}}:{{t.predicate}}="{{t.value}}"
</span>

Hashdd

hashdd.com is a search engine for file hashes which automatically queries 3rd party services like VirusTotal and enriches the information provided based on the 3rd party data. The analyzer includes two flavors: Status and Detail. The first one is used to query hashdd without an API key for the threat level only. The latter produces additional meta information about the sample, but requires an API key.

Results are displayed in TheHive in the following manner:

Hashdd short report

Hashdd status long report

Hashdd detail long report
hashdd — short and long report samples

URLhaus

URLhaus, a service that shares the latest malware download URLs and reports those sites to their respective hosting companies, can now be queried for domains, URLs and hashes. If the observable is found, available information will be displayed as follows:

URLhaus short report

URLhaus long report
URLhaus – short and long report samples

Domaintools Risk and Reputation

New Risk and Reputation services from Domaintools have been added as new flavors to the existing Domaintools analyzer set.

Risk Evidence

The DomainTools Risk Score predicts the risk level and likely threats from a domain that has not been observed in malicious activities, by analyzing various properties of the domain as soon as it is registered.

sc-DTrisk-short

sc-DTrisk-long
Domaintools Risk Evidence – short and long report

Reputation

The Domaintools Reputation Score gives indications about how closely a domain is related to known bad domains, actors, and IPs.

sc-DTReputation-short

sc-DTReputation-long
Domaintools Reputation – short and long report samples

An All New FileInfo

FileInfo performs local static analysis of file observables. It has been completely rewritten from the ground up to be more flexible thus it can easily be enriched with new supported file types and analysis modules. We took this opportunity to merge MsgParser, in charge of extracting and displaying Outlook emails into FileInfo.

As of this release, FileInfo now supports PDF, PE, MS Office documents and Outlook .msg files. We also added support for DDE detection and link extraction in MS Office documents, thanks to Decalage who added this in Oletools since v0.52.

sc-FileInfo-short

sc-FileInfo-long2
FileInfo – short and long report samples

Bug fixes

  • #286 : we updated the way MISP analyzer validates its SSL configuration

  • #292 : we fixed the API URL of malwares.com in the Malwares analyzer

Get It While Supply Lasts!

Each analyzer comes with its own, pip compatible requirements.txt file. To update your Cortex analyzers to 1.11.0, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip2 install -r $I; done && \
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -r $I || true; done

Once done, do not forget to login to Cortex as an orgadmin and click on the Refresh Analyzers button.

Update TheHive Report Templates

If you are using TheHive, get the latest version of  the report templates and import them into TheHive.

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Unveiling Cortex 2

TheHive Project’s Master Chefs are extremely happy to share, for free their latest recipe with the Cyber Threat Intelligence, Digital Forensics and Incident Response communities: Cortex 2.

As its predecessor, Cortex 2 is published under an AGPL v3 license and it introduces many important features that we brushed upon in a Dec 2017 post.

Screen Shot 2017-12-15 at 17.16.06
Cortex 2 — Architecture

Update: Cortex 2.0.1 was released since this post went live. It corrects a few bugs we uncovered in 2.0.0 as described in the changelog. Please install Cortex 2.0.1 instead of 2.0.0.

Authentication

Cortex 2 supports all the authentication methods that TheHive supports: LDAP, Active Directory, local accounts, API Keys, and X.509 SSO.

To connect your favorite Security Incident Response Platform with Cortex 2, you will need to update TheHive to Cerana 0.7 (TheHive 3.0.7) which was released today as well. This version fixes a regression pertaining to case templates introduced by Cerana 0.6 and is the first version to fully support Cortex 2’s API changes and authentication.

To make TheHive 3.0.7 analyze observables at scale through Cortex 2, you have to create an account on Cortex 2 with the read and analyze roles (see the next section) and generate the associated API Key. Next, feed the key in TheHive’s /etc/thehive/application.conf as described in the documentation et voilà !

TheHive 3.0.7 remains compatible with Cortex 1 and you can connect it to a mixed set of Cortex 1 and/or Cortex 2 instances with no issues.

Organizations, Analyzers and Rate Limiting

Cortex 2 introduces multi-tenancy through organizations and each organization can have its own set of users, with different roles, its own set of analyzers and, if necessary, rate limits that will prevent analysts from burning quotas.

Multi-tenancy has several interesting use cases. For instance, if you are the CSIRT or CERT of a large multinational organization with several regional teams, you can create an organization for each region within your constituency and enable the analyzers that they may need to use. Let’s assume that you bought a VirusTotal subscription that limits you to 5000 requests per month. You can configure the corresponding analyzers to give each region a fair share of that quota and keeping some requests for your own use.

In case you are a commercial CSIRT or an MSSP, you could do the same for your customers by installing only one Cortex 2 instance and creating an organization for each customer.

Screen Shot 2018-03-29 at 16.27.05.png
Configure an analyzer graphically and impose rate limits if necessary

User Roles

By default, Cortex 2 is shipped with the default cortex organization which sole purpose is to create other ones and manage the users within each organization and their associated powers. The cortex organization hosts all users with the superAdmin role and it cannot be used to configure or run analyzers.

As described in the new Quick Start Guide, after installing Cortex 2, updating its database and creating the first user who will have super admin powers, you’ll have to create your first organization and at least one user within that organization with orgAdmin rights.

Screen Shot 2018-03-29 at 16.33.02
Create an organization

You can then log out and log in using the orgAdmin account to create further users within that organization, enable and configure analyzers etc. Please note that no analyzer is enabled by default and you need at least v 1.9.0 of the cortex-analyzers repository. To update your set of analyzers to 1.9.0, please run git pull.

Screen Shot 2018-03-29 at 16.28.47
Manage users within an organization

Besides the superAdmin and orgAdmin roles, Cortex 2 introduces the read role which allows users to access analyzer reports and read them but not execute analyzers. For that, users need the analyze role (which implies the read role). orgAdmin users can also run analyzers. superAdmin users are limited to the default cortex organization. While they can create organizations and manage users within them, they cannot access analyzer configurations such as confidential API keys or job reports.

Screen Shot 2018-03-29 at 16.31.28
Job reports

Report Persistence and Caching

Cortex 2 relies on Elasticsearch 5.x to store many configuration items but also all the analyzer reports that have been generated. Unlike its predecessor, you won’t lose your existing reports should you need to restart the service or the host it is running on.

Cortex 2 also introduces report caching. By default the cache.job parameter is set to 10 minutes in /etc/cortex/application.conf. That means that if an analysis on a given observable with a defined TLP is requested and that a report has been previously generated in the last 10 minutes, Cortex 2 will serve that report instead of running a new analysis. This feature can help prevent soliciting analyzers, particularly those which require a subscription or have quotas, when there is no need to do so. Please note that this parameter is global to all the analyzers and all the organizations that are configured in the Cortex 2 instance. We do have plans to make it more granular in future versions.

Migrating from Cortex 1

If you are migrating from Cortex 1.x, we recommend that you:

  1. Save the configuration of your analyzers (which ones are enabled and what their configuration items are, such as users/passwords or API keys).
  2. Install Cortex 2.
  3. Edit /etc/cortex/application.conf to add the secret key as shown in Step 1 of the Quick Start Guide and point Cortex to the location of the analyzers.
  4. Follow the remaining steps of the Quick Start Guide to enable the analyzers you need and reinject their configuration.

What’s Next?

In the upcoming weeks, we will release a new version of Cortex4py in order to make it compatible with Cortex 2, continue the work we started with our MISP Project friends to support MISP attribute enrichment through Cortex 2 (MISP currently only supports enrichment using Cortex 1), and perform a long-overdue overhaul of our documentation.

Feeling Generous? Donate!

As you know, we are a FOSS project and donations are always welcome to make our products even better for the community.

All donations go to Creative Source, the non-profit organization we have created, and we will use them to improve TheHive, Cortex & Hippocampe but also to develop (even better) integrations with other FOSS solutions such as MISP.

So if you are feeling generous, please contact us at support@thehive-project.org.

Creative Source can also provide so-called professional, entreprise-grade support, help integrating the products, train your analysts before they drain or assist you in specific areas such as developing in-house analyzers for Cortex.

Support

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

Correction: March 30, 2018
Instructions on how to update the cortex-analyzers have been added. Also, Cortex 2.0.1 was released to correct a few bugs in the previous version since this post went live.

Cortex Hits the 30 Analyzers Mark

Cortex has now 30 analyzers thanks to Daniil Yugoslavskiy, Davide Arcuri and Andrea Garavaglia (from LDO-CERT) as well as our longtime friend Sébastien Larinier. Their contributions, all under an AGPLv3 license, add handy ways to assess observables and obtain invaluable insight to an already solid Threat Intelligence and DFIR toolset.

In addition to these 3 new analyzers, v 1.7.0 of the Cortex-Analyzers repository also fixes a number of bugs and add a few improvements to existing analyzers as well.

To get the new release, go to your existing Cortex-Analyzers folder and run git pull.

HybridAnalysis

The HybridAnalysis analyzer has been contributed by Daniil Yugoslavskiy. It fetches Hybrid Analysis reports associated with hashes and filenames. This analyzer comes in only one flavor called HybridAnalysis_GetReport.

Requirements

You need to have or create a free Hybrid Analysis account.  Follow the instructions outlined on the Hybrid Analysis API page to generate an API key/secret pair. Provide the API key as a value for the key parameter and the secret as a value to the secret parameter, add the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.

HybridAnalysis {
  secret = "mysecret"
  key = "myAPIKEY"
}

When run from TheHive, the analyzer produces short and long reports such as the following:

sc-short-hybridanalysis_1_0.png

TheHive: HybridAnalysis 1.0 Analyzer – Short and Long Report Samples
TheHive: HybridAnalysis 1.0 Analyzer – Short and Long Report Samples

EmergingThreats

The EmergingThreats analyzer has been submitted by Davide Arcuri and Andrea Garavaglia  from LDO-CERT. It leverages Proofpoint’s Emerging Threats Intelligence service to assess the reputation of various observables and obtain additional and valuable information on malware.

The service comes in three flavors:

  • EmergingThreats_DomainInfo: retrieve ET reputation, related malware, and IDS requests for a given domain.
  • EmergingThreats_IPInfo: retrieve ET reputation, related malware, and IDS requests for a given IP address.
  • EmergingThreats_MalwareInfo: retrieve ET details and info related to a malware hash.

Requirements

You need a valid Proofpoint ET Intelligence subscription.  Retrieve the API key associated with your account and provide it as a value to the key parameter, add the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.

 EmergingThreats {
   key="MYETINTELKEYGOESHERE"
 }

When run from TheHive, it produces short and long reports such as the following:

sc-short-ET_1_0.png

sc-long-ET-1_1_0.png

sc-long-ET-2_1_0.png

sc-long-ET-3_1_0.png

sc-long-ET-4_1_0.png

sc-long-ET-5_1_0.png
TheHive: EmergingThreats 1.0 Analyzer – Short and Long Report Samples

Shodan

The Shodan analyzer is the first submission by Sébastien Larinier. It lets you retrieve key Shodan information on domains and IP addresses.

This analyzer comes in two flavors:

  • Shodan_Host: get Shodan information on a host.
  • Shodan_Search: get Shodan information on a domain.

Requirements

You need to create a Shodan account and retrieve the associated API Key. For
best results, it is advised to get a Membership level account, otherwise a free one can be used.

Supply the API key as the value for the key parameter, add the lines below to the config section of /etc/cortex/application.conf then restart the cortex service.

Shodan {
  key= "myawesomeapikey"
}

When run from TheHive, it produces short and long reports such as the following:

sc-short-shodan_1_0.png

sc-long-shodan_1_0.png
TheHive: Shodan 1.0 Analyzer – Short and Long Report Samples

Miscellaneous Fixes and Improvements

  • #100 : support both Cuckoo versions – by Garavaglia Andrea
  • #113 : Cuckoo Analyzer requires final slash – by Garavaglia Andrea
  • #93 : VirusTotal URL Scan Bug
  • #101 : Missing olefile in MsgParser requirements
  • #126 : PhishTank analyzer doesn’t work – by Ilya Glotov

Update TheHive Report Templates

If you are using TheHive, get the last version of  the report templates and import them into TheHive.

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Cortex 1.1.4 Released

Moments ago, we have announced the release of Mellifera 13, TheHive4py 1.3.0, and Cortex4py. And since we don’t want to leave you wanting for more fun time, you may want to schedule as well a Cortex update shall you need it 😉

Implemented Enhancements

  • Disable analyzer in configuration file #32
  • Group ownership in Docker image prevents running on OpenShift #42

Fixed Bugs

  • Cortex removes the input details from failure reports #38
  • Display a error notification on analyzer start fail #39

Download & Get Down to Work

To update your current Cortex installation, follow the instructions of the installation guide. Before doing so, you may want to save the job reports that were not executed via TheHive. Cortex 1 has no persistence and restarting the service will wipe out any existing reports.

Please note that you can install Cortex using an RPM or DEB package, deploy it using an Ansible script, use Docker, install it from a binary or build it from sources.

Support

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

TheHive, Cortex and MISP: How They All Fit Together

TheHive, Cortex and MISP work nicely together and if you’ve read our June-Dec 17 roadmap post, the integration of our products with the de facto threat sharing platform will get better in a few months.

During the FIRST conference presentation we gave last week, we displayed a picture that we will use here to try to explain how these three open source and free products integrate with one another.

Screen Shot 2017-06-16 at 09.58.43.png
A Picture is Worth a Thousand Words…

TheHive

TheHive is a Security Incident Response Platform (SIRP). It can receive alerts from different sources (SIEM, IDS, email. etc.) via its REST API. This is where alert feeders come into play.

Alert Feeders

Think of an alert feeder as a specialized program which consumes a security event (SIEM alert, email report, IDS alert, and so on), parses it and outputs an alert that its sends to TheHive through TheHive4py, the Python library we provide to interact with TheHive’s REST API.

We do not supply such feeders but developing them should be straightforward. If not, let us know  and we’ll do our best to help you out.

Alerts

Any alert sent to TheHive will show up in its Alerts pane. In addition to the sources mentioned above, new or updated MISP events will show up as well in that area if you configured TheHive to connect to one or several MISP instances. If so, TheHive will poll those MISP instance(s) at every interval looking for new or updated events. If there are any, TheHive will generate an alert which will end up in the Alerts pane.

Screen Shot 2017-06-18 at 15.29.51.png
The Alerts Pane

Alerts can be ignored, mark as read, previewed and imported. When an alert is imported, it becomes a case that needs to be investigated.

Cases

workflow
The Workflow that is at the Heart of TheHive

A case can be generated from an alert or created from scratch. It is subdivided into tasks (think identification, containment, eradication, check proxy logs, and so on) and observables (IP addresses, hashes, email addresses, domain names, URLs…). When analysts are working on tasks, they add logs as they go. In TheHive’s terminology, logs are text entries which may contain attachments to help analysts record what they have been doing. Logs can be written using Markdown or a rich-text editor.

Case Templates

You don’t need to add the same tasks over and over when working on cases belonging to a given category (DDoS, Malspam, APT, …). You can create custom templates to which you add tasks as shown below. This is very useful when you are dealing with alerts so that when you import them, you can select which case template you’d like to apply and there you go!

Screen Shot 2017-06-16 at 10.26.22.png
A Sample Case Template

Observables

Observables can be tagged, flagged as IOCs, and analyzed. When the investigation is well in progress or completed, you may want to share the resulting IOCs or a subset of those with partners and peers. TheHive will support the ability to export that data to MISP in September 2017. Until then, you can still export your IOCs as text, CSV or as a MISP-compatible format that you can use to add them to your MISP instance using the freetext editor. TheHive can export IOCs/observables in protected (hxxps://www[.]somewhere[.]com/) or unprotected mode.

Every observable must have a TLP (Traffic Light Protocol) level. By default, any added observable is considered TLP:AMBER. Please note that the TLP is taken into account by some analyzers. Wait! Analyzers?

Cortex

Cortex is our standalone analysis engine and a perfect companion for TheHive and MISP. Analysts can use it to analyze observables using its Web UI, in which case they can be submitted only one at a time. The Web UI should really be limited to quick assessments of observables before creating a case in TheHive (or in an alternate SIRP). The power of Cortex really comes into play when you use its REST API. TheHive speaks natively to Cortex (as MISP does). Moreover, TheHive can leverage one or several Cortex servers.

Screen Shot 2017-06-18 at 15.57.27.png
Observable Page and List of Analyzers

Analyzers

As of this writing, Cortex has 23 analyzers which come in a total of 39 flavors and more will be available soon.

An analyzer can be written in any programming language supported by Linux though all of our current analyzers are written in Python. This is because we provide a Python library called Cortexutils which contains a set of utility classes that make it easier to write an analyzer in Python.

Flavors

Analyzers such as VirusTotal, PassiveTotal or DomainTools can provide different analysis services. Let’s take VirusTotal as an example. You can scan a file or URL. That’s one flavor. You can also obtain the latest available report on VirusTotal.com for a file, hash, domain or IP address. That’s a second flavor. So the VirusTotal analyzer has two flavors.

Screen Shot 2017-06-18 at 16.26.41.png

How about PassiveTotal? It has 8 flavors: unique resolutions lookup, SSL certificate history lookup, malware lookup, passive DNS lookup, data enrichment lookup, SSL certificate details lookup, OSINT lookup and WHOIS data lookup.

The MISP Search Analyzer

At this point, we need to mention a special analyzer that may create some confusion if not understood correctly: the MISP Search analyzer. Thanks to it, Cortex has the ability to search observables within a MISP instance as represented by the arrow that goes from the Analyzers to MISP.

Screen_Shot_2017-06-19_at_08_03_54.png
Search for MISP Events Containing a Given Observable

When an observable is found in an event, Cortex will return the number of records found (i.e. the number of events where the observable has been found) and a list of links to those events with additional data.

Screen_Shot_2017-06-19_at_08_13_16.png
Searching for a Hash Using the MISP Search Analyzer from the Cortex Web UI

Screen Shot 2017-06-19 at 08.17.04.png
The Same Search Conducted from TheHive: Long Report

Screen Shot 2017-06-19 at 08.18.58.png
Mini-Report

The current version of the MISP Search analyzer can only search within a single MISP instance but in the near future, it will be able to support multiple ones.

MISP Expansion Modules

Besides its own analyzers (which include MISP Search described above), Cortex can also invoke MISP expansion modules. These are normally used by MISP to enrich attributes within events but Cortex can also take advantage of them to analyze observables.

There is some overlap between the native Cortex analyzers and MISP expansion modules. When choosing between a native analyzer or an expansion module, we highly recommend you select the former. The expansion modules are deactivated in the default Cortex configuration.

Jobs

When you submit an observable for analysis, Cortex will create a job and, if successful, it will generate an analysis report in JSON format. TheHive has the ability to parse those results and present them in a human-friendly fashion thanks to report templates we offer for free. So when you’ll submit an observable to Cortex from TheHive, you’ll get back a short (or mini) report and a long one. The first can be thought of as a really tiny Exec Analyst Summary while the second provides more insight and details.

Calling Cortex from MISP

In addition to the expansion modules we have just mentioned, MISP 2.4.73 and up can enrich attributes using Cortex analyzers. The configuration is pretty straightforward. So if all you are concerned about is threat intelligence and sharing, you may augment your visibility into a given threat represented as a MISP event by leveraging all current 23 Cortex analyzers and any future ones.

Conclusion

TheHive, Cortex and MISP are three open source and free products that can highly aid you combat threats and keep the ‘monsters’ at bay.

TheHive, as a SIRP, allows you to investigate security incident swiftly in a collaborative manner. Several analysts can work simultaneously on tasks & cases . While cases can be created from scratch, TheHive can receive alerts from different sources thanks to alert feeders which consume security events generated by multiple sources and feed them into TheHive using TheHive4py Python library. TheHive can also sync to one or several MISP instances to receive new and updated events which will appear in the alert pane with all the other alerts generated by other sources. Analysts can then preview new alerts to decide whether they need to be acted upon. If so, they can transform them into investigation cases using templates.

To analyze the observables collected in the course of an investigation and/or imported from a MISP event, TheHive can rely on one or several Cortex analysis engines. Cortex is another standalone product that we have developed which sole purpose is to allow you to analyze observables at scale thanks to its large number of analyzers, MISP expansion modules and any analyzer you might have developed on the side. Cortex has a REST API that can be used to empower other security products such as  ‘analytics’ software, alternate SIRPs or MISP.

The highly popular threat sharing platform can indeed enrich attributes thanks to Cortex as it has a native integration with it. And in a few months, you will also be able to export cases from TheHive as MISP events that you can share with peers and partners.

If you do share, you do care about our collective mission to defend the  digital assets that are under our watch from harm. So let us fight together as one.

 

 

Joe Sandbox, MISP Search and Report Improvements

We are thrilled to announce that Cortex has two new analyzers: Joe Sandbox and MISP Search. Moreover, we have produced new analyzer report templates for TheHive and improved existing ones.

Joe Sandbox

List JSB Cortex.png
Cortex: New Joe Sandbox Analyzer

Joe Sandbox, by Joe Security LLC, is a very powerful malware analysis platform that has been around for many years and comes in two flavors: cloud and on-premises. The Joe Sandbox Cortex analyzer has been tested using an on-prem Joe Sandbox Ultimate version and can process URLs and files. The analyzer can process files with or without Internet access.

To use the analyzer, you must provide the API key of your Joe Sandbox instance. You must log in to Joe Sandbox, click on your account name, then on Settings and on the API Key tab.

report JSB Cortex.png
Cortex: Joe Sandbox Output Example

We have produced a report template for the Joe Sandbox analyzer output resulting from file analysis. The URL analysis report template is not yet available but it should be in a few days.

JSB TH short report

JSB_THEHIVE.png
TheHive: Joe Sandbox Analyzer – Short and Long Report Samples

MISP Search

Screen Shot 2017-03-31 at 15.25.07.png
Cortex: New MISP Search Analyzer

It is no longer necessary to present MISP, the de facto standard of threat sharing. The new MISP Search analyzer will let you search events containing the observable you provide as an input. It applies to a lot of observable types as you can see in the screenshot above.

To use it, you’ll need to supply the API key available in the MISP UI interface.

result_MISP_Cortex.jpg
Cortex: MISP Analyzer Output Sample

Nils Kuhnert created an alternate MISP Search analyzer which has the ability to query multiple MISP instances. We are currently reviewing his submission along with several other analyzers he contributed before improving the newly released MISP Search analyzer.

PassiveTotal Report Templates

PT Whois short report.pngPT UniqueRes TH short report.png

While we published the PassiveTotal analyzer weeks ago, TheHive didn’t have report templates for it at the time. We have now new, shiny short and long report templates for most of the services provided by the PT analyzer.

PT PDNS long report.png
TheHive: PassiveTotal PassiveDNS – Long Report Sample

DomainTools Whois Lookup Report Template

DT Whois TH short report.png

The short report templates of the DomainTools Whois Lookup analyzer has been improved. We now use a taxonomy to provide more context and differentiate between the DomainTools and PassiveTotal Whois results.

VirusTotal Get Report and VirusTotal Scan Report Templates

VT TH short report.png
VT and JSB TH short report.png

The short report templates for both services have also been improved to use a taxonomy to provide additional context and distinguish their results from the PassiveTotal Malware service.

Get the new analyzers

To install the new analyzers, grab the Cortex-Analyzers repository and unpack its content (or git pull the master  branch) in your existing /path/to/cortex-analyzers.

The Joe Sandbox analyzer does not need any additional Python library if you have already installed Cortex and the analyzers following the guide we provide.  To use it, edit your Cortex configuration file (/path/to/cortex/application.conf) and add the following lines in the analyzer section:

 JoeSandbox {
     apikey="..."
     url="..."
 }

By default, Joe Sandbox will time out the analysis after 30*60 seconds (30 minutes). Additionally, the analyzer will wait for the Joe Sandbox server to respond within 30 seconds. If no response is received within this period, it will time out. If you want to override these values, you’ll need to add the following lines in the analyzer section:

JoeSandbox {
     apikey="..."
     url="..."
     analysistimeout=<NEW VALUE> # optional
     networktimeout=<NEW VALUE> # optional
}

The MISP Search analyzer requires pymisp. Use the following command line to install the required library:

sudo pip install pymisp

Then edit your Cortex configuration file (/path/to/cortex/application.conf) and add the following lines in the analyzer section:

MISP {
     api_key="..."
     url="..."
}

Please note that you must restart Cortex to take the changes into account. The current version has no persistence so you’ll lose all your existing jobs.

You can find the full installation requirements for Cortex and Cortex-Analyzers on the Cortex wiki pages.

Use the New Report Templates

To import the new report templates in your instance of TheHive:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help you!