Cortex-Analyzers 2.7.0: 5 Analyzers, 1 Responder

Good morning (or evening if you are on that side of the planet) folks!

We had a very busy week, packed with announcements. First, we released TheHive 4.0-RC2 which you’ve certainly taken to test, then we announced two patch releases for TheHive 3.4. And guess what? Here are some additional Cortex analyzers, a responder and a number of fixes and improvements for existing ones, bringing the total to a whopping 146 analyzers and 18 responders!

New Analyzers

New Responders

Analyzers

ANY.RUN

ANY.RUN is a malware sandbox service in the cloud. By using this analyzer, an analyst can submit a suspicious file or URL to the service for analysis and get a report. The report can contain various information such as:

  • Interactive access
  • Research threats by filter in public submissions
  • File and URL dynamic analysis
  • Mitre ATT&CK mapping
  • Detailed malware reports
ANY.RUN short report
ANY.RUN long report

CyberChef

CyberChef is a simple, intuitive web app for carrying out all manner of “cyber” operations within a web browser. These operations include simple encoding like XOR or Base64, more complex encryption like AES, DES and Blowfish, creating binary and hexdumps, compression and decompression of data, calculating hashes and checksums, IPv6 and X.509 parsing, changing character encodings, and much more.

This analyzer connects to a CyberChef-server and comes in 3 flavors:

  • CyberChef_FromBase64, that takes Base64 strings as input for CyberChef-server
  • CyberChef_FromCharCode, that takes CharCode as input for CyberChef-server and run this recipe
  • CyberChef_FromHex, that takes Hex strings as input for CyberChef-server

TheHive displays the analyzer results as follows:

CyberChef short report
image
CyberChef long report

MalwareBazaar

MalwareBazaar is a project from abuse.ch with the goal of sharing malware samples with the infosec community, AV vendors and threat intelligence providers.

This analyzer allows analysts to query the API of this service on observables of types ip, domain, fqdn, url, and hash.

TheHive displays the analyzer results as follows:

MalwareBazaar short report
MalwareBazaar long report

OpenCTI

OpenCTI is an open source platform allowing organisations to manage their Cyber Threat Intelligence knowledge and observables. It has been created in order to structure, store, organise and visualise technical and non-technical information about cyber threats.

This analyzer allows an analyst to query the API and request for information about observables of types domain, ip, url, fqdn, uri_path, user-agent, hash, email, mail, mail_subject, registry, regexp, filename and other.

TheHive displays the analyzer results as follows:

OpenCTI short report
OpenCTI long report

MISPWarningLists reloaded (need for speed aka DB support)

The previous version of this analyzer basically used to clone the MISP Warning lists repository and do a lookup in downloaded files. This can be very long to complete.

This new version introduces the optional support of PostgreSQL:

  • To store warning lists, in a similar way to the NSRL (National Software Reference Library) Analyzer.
  • Make lookups through these lists faster.

If you want to benefit from the performance boost, using a PostgreSQL server to store the data, you can simply install the requirements.txt, feed the database and configure the connection in the configuration as well:

  • First, sync with the misp-warninglists GitHub repository
  • In the analyzer folder, use the program warninglists_create_db.py to import the warning lists in PostgreSQL. Before running, edit the program file and update the path of where your lists are stored (warninglists_path = "misp-warninglists/**/list.json")
  • You can schedule these jobs (ex. with cron): first, sync a folder with the repository, and then run the program to update the database.

Once done, configure the analyzer with the conn parameter to connect to the database, or, if you prefer to continue using the previous behaviour and do your lookups in files, just specify the path of the folder:

MISPWarningList Configuration example

Templates have also been updated, and TheHive displays the analyzer results as follows:

MISPWarningList short report
MISPWarningList long report

Responders

RT4-CreateTicket

RT4 (Request Tracker) is a ticketing system. With this responder, an analyst can create a ticket in RT. CaseID is submitted to RT as a reference.

Unfortunately, like for some other analyzers and responders, we have not been able to test this responder on our side. Please feel free to share your feedback with us and also with Michael Davis, who we would like to thank for the hard work and for having shared this responder with the community.

Fixes and Improvements

  • Fix Inconsistent Key References in Shodan Analyzer (#748)
  • Fix SSL & python3 for Yeti Analyzer (#468 , #708)
  • Fix bug in Emlparser Analyzer (#730)
  • Fix in Shodan Analyzer: Inconsistent Key References (#748)
  • Support python3 in DNSDB Analyzer (#613)
  • Support APIKey for EmailRep Analyzer (#750)
  • Improvement: EmlParser now extracts some useful IOCs (#710)

Get It While Supply Lasts!

If you are still using the old-style way of installing analyzers and responders, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done
for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates` menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Cortex-Analyzers 2.6.0: 146 Analyzers, 18 Responders

Amidst the ongoing COVID-19 crisis, we managed to release Cortex-Analyzers 2.6.0, which includes 4 new Analyzers, 2 new Responders, and a large number of bug fixes and improvements.

We’d like to thank all the contributors for their awesome work!

We truly appreciate the time they generously give away for helping our fellow cyberdefenders out there protect their environments against attackers who are also in lockdown mode. Attackers who, instead of playing board games or chess, are playing with our nerves and the hordes of teleworkers who are willing to click on anything that provide the ‘latest and greatest COVID-19 information’ or which can help them do their jobs (like this wonderful ‘Zoon’ video-conferencing application 😋).

Les Temps modernes - Film (1936) - SensCritique
Source: senscritique.com

What’s New?

New Analyzers

New Responders

Analyzers

DomainTools Iris

The Investigate flavour was missing from the DomainToolsIris analyzer that was included in Cortex-Analyzers 2.4.0. This is now fixed. This new flavour can be used to gather interesting information on a domain.

TheHive displays the analyzer results as follows:

DomainToolsIris_Investigate short reports
DomainToolsIris_Investigate long report

IntezerCommunity

Intezer Analyze™ is a cloud-based malware analysis service that provides an extensive understanding of any executable file by comparing code on a massive scale to a comprehensive database of malware and trusted software. 

This analyzer can be used to submit a file to the Intezer service for analysis.

TheHive displays the analyzer results as follows:

IntezerCommunity short report
IntezerCommunity long report

NSRL

The National Software Reference Library (NSRL) is designed to collect software from various sources and incorporate file profiles computed from this software into a Reference Data Set (RDS) of information. The RDS can be used by law enforcement, government, and industry organisations to review files on a computer by matching file profiles in the RDS. This will help alleviate much of the effort involved in determining which files are important as evidence on computers or file systems that have been seized as part of criminal investigations.

In order to use this analyzer, you must download and extract NSRLFile files from the NIST website. You can pick multiple files but you need to rename them in order to understand which file contains the required information.

All files are called NSRLFile.txt, renaming them permit to understand in which file the record has been found.

The analyzer can operate in 2 different ways with 2 completely different performance profiles (we’re speaking around 30 secs vs 0.05 sec):

  1. lookup in plain files
  2. lookup in a database

If you are planning to use this analyzer for many searches, then the second option is suggested and we provide a script to help you parse, validate and insert data in a PostgreSQL database. If you choose this option, consider that the DB size can be around 4 times bigger than plain files.

NSRL Lookup short template
NSRL Lookup long report

UrlScan.io

The URLScan.io analyzer has been updated with a new Scan flavour. Until now, this analyzer allowed to request report regarding a url, domain, fqdn observable. With this new flavour, anyone with a valid API key, which can be obtained for free, can request a scan on observables of the same type.

UrlScan.io short template
UrlScan.io long template

Responders

DomainToolsIris_CheckMaliciousTags

Depending on the reports generated by the DomainToolsIris analyzer, this responder adds a tag at the Case and Observable level if something malicious is found. This responder can be updated to add more custom actions depending on your needs and environment.

DomainToolsIris_AddRiskyDNSTag

Depending on on the reports generated by the DomainToolsIris analyzer, this responder adds a tag at the Case and Observable level if one of the domain observables is considered risky. This responder can be updated to add more custom actions depending on your needs and environment.

Fixes and Improvements

  • Improve TalosReputation analyzer (#521)
  • MISP WarningList analyzer fixed (#538)
  • Error fixed in ThreatCrowd (#518)
  • Encoding related bug fixed in Mailer 1_0 (#573)
  • API has changed: temporary fix for Crt_sh_Transparency_Logs_1_0 (#594)
  • Analyzers missing cortexutils in requirements (#695)
  • New mime types for Office documents in FileInfo (#705)
  • UmbrellaBlacklister analyzer now support fqdn and url observables (#547)
  • URLHaus analyzer support fqdn observables (#556)
  • Abuselpdb now support APIv2 (#618)

Get It While Supply Lasts!

If you are still using the old-style way of installing analyzers and responders, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done
for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates` menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Cortex-Analyzers 1.15.3 get ready for URLhaus and Cuckoo

Few improvements have been introduced in this version :

  • Proofpoint analyzer has been updated to use python3 (#417)
  • Long report of Cuckoo Sandbox analyzer has been improved to be able to display Cuckoo v. 2.0.6 reports (#418)
  • URLhaus analyzer has also been updated to use the recently introduced API (#431)

On your Cortex server, update your analyzers with the following command:

$ cd /opt/Cortex-Analyzers && git pull
$ for I in analyzers/*/requirements.txt; do sudo -H /usr/bin/python2 -m pip install -r $I
$ for I in analyzers/*/requirements.txt; do sudo -H /usr/bin/python3 -m pip install -r $I

Updating analyzers should be followed by also updating report templates. Download new report templates and add the archive in TheHive report templates.

Detailed information

Cuckoo Sandbox

Thanks to @nicpenning Cuckoo analyzer is now able to display reports from version  2.0.6 of Cuckoo. With this version, remote connections part of the report has been fixed and is now well displayed

The new cuckoo analyzer template

Notice: This fix has been reviewed by our core team, unfortunately, we have not been able to test it.

URLhaus

Abuse.ch, the operator of URLhaus recently introduced a new API for their service in order to handle bulk queries and reduce costs for their main page that uses CDNs for delivery. Because of that, thanks to Nils Kuhnert, the URLhaus analyzer has been rewritten from scratch. Also, the templates changed because there are three different API endpoints used:

The short reports now display the threat type – which currently is always malware_download and, for the hash observable type, the malware family – if given by URLhaus:

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Keep Them Coming: Hashdd, URLhaus & a Revamped FileInfo analyzer

We are happy to announce the immediate availability of a new major version of Cortex-Analyzers. Version 1.11.0 includes two brand new analyzers, several updates and a few bug fixes:

  • Hashdd, contributed by  iosonogio
  • URLhaus, contributed by  ninoseki
  • The Domaintools analyzer has been updated with two new flavors: Risk and Reputation
  • The VirusTotal analyzer can (finally!) get reports for URL observables
  • MsgParser and File_Info have been merged in a new, shiny, completely rewritten FileInfo analyzer

As we are approaching the 90 analyzers mark, we wholeheartedly thank our user community for continuously contributing new analyzers, testing them and helping us improve the existing ones.

Important Notice

We made significant changes in this release in the analyzers and short reports. Prior to Cortex-Analyzers 1.11.0, the summary() function in the analyzer code generates a result such as:

"taxonomies": [{
     "level": "suspicious",
     "namespace": "DT",
     "predicate": "Risk",
     "value": "\"10\""
 }]

Double quotes were included in the resulting value. We decided to update the summary() function and make it generate the same result without double quotes :

"taxonomies": [{
     "level": "suspicious",
     "namespace": "DT",
     "predicate": "Risk",
     "value": "10"
 }]

To display the double quotes in TheHive, we updated all short reports to include them:

<span class="label" ng-repeat="t in content.taxonomies" ng-class="{'info': 'label-info', 'safe': 'label-success', 'suspicious': 'label-warning', 'malicious':'label-danger'}[t.level]">
     {{t.namespace}}:{{t.predicate}}="{{t.value}}"
</span>

Hashdd

hashdd.com is a search engine for file hashes which automatically queries 3rd party services like VirusTotal and enriches the information provided based on the 3rd party data. The analyzer includes two flavors: Status and Detail. The first one is used to query hashdd without an API key for the threat level only. The latter produces additional meta information about the sample, but requires an API key.

Results are displayed in TheHive in the following manner:

Hashdd short report

Hashdd status long report

Hashdd detail long report
hashdd — short and long report samples

URLhaus

URLhaus, a service that shares the latest malware download URLs and reports those sites to their respective hosting companies, can now be queried for domains, URLs and hashes. If the observable is found, available information will be displayed as follows:

URLhaus short report

URLhaus long report
URLhaus – short and long report samples

Domaintools Risk and Reputation

New Risk and Reputation services from Domaintools have been added as new flavors to the existing Domaintools analyzer set.

Risk Evidence

The DomainTools Risk Score predicts the risk level and likely threats from a domain that has not been observed in malicious activities, by analyzing various properties of the domain as soon as it is registered.

sc-DTrisk-short

sc-DTrisk-long
Domaintools Risk Evidence – short and long report

Reputation

The Domaintools Reputation Score gives indications about how closely a domain is related to known bad domains, actors, and IPs.

sc-DTReputation-short

sc-DTReputation-long
Domaintools Reputation – short and long report samples

An All New FileInfo

FileInfo performs local static analysis of file observables. It has been completely rewritten from the ground up to be more flexible thus it can easily be enriched with new supported file types and analysis modules. We took this opportunity to merge MsgParser, in charge of extracting and displaying Outlook emails into FileInfo.

As of this release, FileInfo now supports PDF, PE, MS Office documents and Outlook .msg files. We also added support for DDE detection and link extraction in MS Office documents, thanks to Decalage who added this in Oletools since v0.52.

sc-FileInfo-short

sc-FileInfo-long2
FileInfo – short and long report samples

Bug fixes

  • #286 : we updated the way MISP analyzer validates its SSL configuration

  • #292 : we fixed the API URL of malwares.com in the Malwares analyzer

Get It While Supply Lasts!

Each analyzer comes with its own, pip compatible requirements.txt file. To update your Cortex analyzers to 1.11.0, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip2 install -r $I; done && \
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -r $I || true; done

Once done, do not forget to login to Cortex as an orgadmin and click on the Refresh Analyzers button.

Update TheHive Report Templates

If you are using TheHive, get the latest version of  the report templates and import them into TheHive.

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!