Cortex-Analyzers 2.8.0: to infinity and beyond!

Thanks to the community and all the contributors, this release comes with 1 new Analyzer, 2 new Responders, lots of improvements and bug fixes.

But there is more news from the front.

Starting from this milestone, bugfixes and new Analyzers or Responders should be released in a smoother way as we are improving few processes. Some changes and recommandations should appear in the next days for submission, and our release process will be improved to fix bugs easier and release new code faster.

We also plan to offer a better documentation. We already started to publish information regarding each Analyzer and Responder. This is a work in progress, and it will be updated with the current requirements guide.

DomainToolsIris documentation page

For each Analyzer and Responder, a page details the purpose of each flavors, the configuration required and even some screenshots from report samples. It will also allow developers to share their own notes if wanted.

New Analyzers

New Responders

Improvements

  • Refactor Onyphe using new v2 api (#736)
  • Improvement in Shodan: add vulns in template and taxonomies (#772 & #776)
  • Improvement in Mailer responder: tasks support and auth (#764, #737, #379)
  • Improvement in SinkDb: support for new api with new dataTypes supported (#483, #498, #756)

Analyzers

LastInfoSec

LastInfoSec offers innovative and automated solutions to collect data, refine it and turn it into useful and actionable information, quickly available to improve the protection, detection and investigation capabilities of companies and government organizations.

TheHive displays the analyzer results as follows:

Short template for LastInfoSec Report
Long Template for LastInfoSec Report

Onyphe

An important work has been made on Onyphe Analyzer to support APIv2. All 7 flavors from older version have been removed and merged into only one flavor named “Onyphe_Summary”. An API key is still needed to query Onyphe API.

TheHive displays the analyzer results as follows:

Onyphe_Summary short report
Onyphe_Summary long report

Responders

Sendgrid

Sendgrid is a customer communication platform for transactional and marketing email used when you have to ensure that your notifications and transactional emails are delivered quickly and securely.

This analyzer works like the Mailer one, but relying on SendGrid external service to delivery emails.

In order to use the service please follow the instruction being careful to the verify your email address.

VirusTotalDownloader

This responders runs on Observables of type “hash” and allows analyst to download corresponding file from VirusTotal. Once downloaded, the file is added to the current case observables in TheHive.

In order to use this responder, a Premium API key from VirusTotal is needed. An API key from TheHive is also needed to upload the file in the observables list.

Use the responder on the hash to add the sample in your Observables

Fixes and Improvements

  • Fix: some analyzer uses invalid “email” dataType (#799)
  • Fix in MalwareBazaar: wrong dataTypes in config (#794)
  • Fix in PhishTank: the JSON object must be str, not ‘bytes’ (#786)
  • Fix in VMRay: fix error in parsing and workflow (#785 & #784)
  • Fix in Wazuh: ipaddress import missing (#778)
  • Fix in Minemeld Responder: requests missing in requirements (#774)
  • Fix in WOT: moving to new endpoint (#771)
  • DomainTools Iris: update api urls (#760)
  • Fix in ThreatResponse: module_type key removed from response (#759)
  • Fix in Abuse_Finder: pythonwhois dependency (#742)

Get It While Supply Lasts!

If you are still using the old-style way of installing analyzers and responders, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done
for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates` menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Cortex-Analyzers 2.7.0: 5 Analyzers, 1 Responder

Good morning (or evening if you are on that side of the planet) folks!

We had a very busy week, packed with announcements. First, we released TheHive 4.0-RC2 which you’ve certainly taken to test, then we announced two patch releases for TheHive 3.4. And guess what? Here are some additional Cortex analyzers, a responder and a number of fixes and improvements for existing ones, bringing the total to a whopping 146 analyzers and 18 responders!

New Analyzers

New Responders

Analyzers

ANY.RUN

ANY.RUN is a malware sandbox service in the cloud. By using this analyzer, an analyst can submit a suspicious file or URL to the service for analysis and get a report. The report can contain various information such as:

  • Interactive access
  • Research threats by filter in public submissions
  • File and URL dynamic analysis
  • Mitre ATT&CK mapping
  • Detailed malware reports
ANY.RUN short report
ANY.RUN long report

CyberChef

CyberChef is a simple, intuitive web app for carrying out all manner of “cyber” operations within a web browser. These operations include simple encoding like XOR or Base64, more complex encryption like AES, DES and Blowfish, creating binary and hexdumps, compression and decompression of data, calculating hashes and checksums, IPv6 and X.509 parsing, changing character encodings, and much more.

This analyzer connects to a CyberChef-server and comes in 3 flavors:

  • CyberChef_FromBase64, that takes Base64 strings as input for CyberChef-server
  • CyberChef_FromCharCode, that takes CharCode as input for CyberChef-server and run this recipe
  • CyberChef_FromHex, that takes Hex strings as input for CyberChef-server

TheHive displays the analyzer results as follows:

CyberChef short report
image
CyberChef long report

MalwareBazaar

MalwareBazaar is a project from abuse.ch with the goal of sharing malware samples with the infosec community, AV vendors and threat intelligence providers.

This analyzer allows analysts to query the API of this service on observables of types ip, domain, fqdn, url, and hash.

TheHive displays the analyzer results as follows:

MalwareBazaar short report
MalwareBazaar long report

OpenCTI

OpenCTI is an open source platform allowing organisations to manage their Cyber Threat Intelligence knowledge and observables. It has been created in order to structure, store, organise and visualise technical and non-technical information about cyber threats.

This analyzer allows an analyst to query the API and request for information about observables of types domain, ip, url, fqdn, uri_path, user-agent, hash, email, mail, mail_subject, registry, regexp, filename and other.

TheHive displays the analyzer results as follows:

OpenCTI short report
OpenCTI long report

MISPWarningLists reloaded (need for speed aka DB support)

The previous version of this analyzer basically used to clone the MISP Warning lists repository and do a lookup in downloaded files. This can be very long to complete.

This new version introduces the optional support of PostgreSQL:

  • To store warning lists, in a similar way to the NSRL (National Software Reference Library) Analyzer.
  • Make lookups through these lists faster.

If you want to benefit from the performance boost, using a PostgreSQL server to store the data, you can simply install the requirements.txt, feed the database and configure the connection in the configuration as well:

  • First, sync with the misp-warninglists GitHub repository
  • In the analyzer folder, use the program warninglists_create_db.py to import the warning lists in PostgreSQL. Before running, edit the program file and update the path of where your lists are stored (warninglists_path = "misp-warninglists/**/list.json")
  • You can schedule these jobs (ex. with cron): first, sync a folder with the repository, and then run the program to update the database.

Once done, configure the analyzer with the conn parameter to connect to the database, or, if you prefer to continue using the previous behaviour and do your lookups in files, just specify the path of the folder:

MISPWarningList Configuration example

Templates have also been updated, and TheHive displays the analyzer results as follows:

MISPWarningList short report
MISPWarningList long report

Responders

RT4-CreateTicket

RT4 (Request Tracker) is a ticketing system. With this responder, an analyst can create a ticket in RT. CaseID is submitted to RT as a reference.

Unfortunately, like for some other analyzers and responders, we have not been able to test this responder on our side. Please feel free to share your feedback with us and also with Michael Davis, who we would like to thank for the hard work and for having shared this responder with the community.

Fixes and Improvements

  • Fix Inconsistent Key References in Shodan Analyzer (#748)
  • Fix SSL & python3 for Yeti Analyzer (#468 , #708)
  • Fix bug in Emlparser Analyzer (#730)
  • Fix in Shodan Analyzer: Inconsistent Key References (#748)
  • Support python3 in DNSDB Analyzer (#613)
  • Support APIKey for EmailRep Analyzer (#750)
  • Improvement: EmlParser now extracts some useful IOCs (#710)

Get It While Supply Lasts!

If you are still using the old-style way of installing analyzers and responders, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done
for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates` menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Cortex-Analyzers 2.6.0: 146 Analyzers, 18 Responders

Amidst the ongoing COVID-19 crisis, we managed to release Cortex-Analyzers 2.6.0, which includes 4 new Analyzers, 2 new Responders, and a large number of bug fixes and improvements.

We’d like to thank all the contributors for their awesome work!

We truly appreciate the time they generously give away for helping our fellow cyberdefenders out there protect their environments against attackers who are also in lockdown mode. Attackers who, instead of playing board games or chess, are playing with our nerves and the hordes of teleworkers who are willing to click on anything that provide the ‘latest and greatest COVID-19 information’ or which can help them do their jobs (like this wonderful ‘Zoon’ video-conferencing application ūüėč).

Les Temps modernes - Film (1936) - SensCritique
Source: senscritique.com

What’s New?

New Analyzers

New Responders

Analyzers

DomainTools Iris

The Investigate flavour was missing from the DomainToolsIris analyzer that was included in Cortex-Analyzers 2.4.0. This is now fixed. This new flavour can be used to gather interesting information on a domain.

TheHive displays the analyzer results as follows:

DomainToolsIris_Investigate short reports
DomainToolsIris_Investigate long report

IntezerCommunity

Intezer Analyze‚ĄĘ is a cloud-based malware analysis service that provides an extensive understanding of any executable file by comparing code on a massive scale to a comprehensive database of malware and trusted software. 

This analyzer can be used to submit a file to the Intezer service for analysis.

TheHive displays the analyzer results as follows:

IntezerCommunity short report
IntezerCommunity long report

NSRL

The National Software Reference Library (NSRL) is designed to collect software from various sources and incorporate file profiles computed from this software into a Reference Data Set (RDS) of information. The RDS can be used by law enforcement, government, and industry organisations to review files on a computer by matching file profiles in the RDS. This will help alleviate much of the effort involved in determining which files are important as evidence on computers or file systems that have been seized as part of criminal investigations.

In order to use this analyzer, you must download and extract NSRLFile files from the NIST website. You can pick multiple files but you need to rename them in order to understand which file contains the required information.

All files are called NSRLFile.txt, renaming them permit to understand in which file the record has been found.

The analyzer can operate in 2 different ways with 2 completely different performance profiles (we’re speaking around 30 secs vs 0.05 sec):

  1. lookup in plain files
  2. lookup in a database

If you are planning to use this analyzer for many searches, then the second option is suggested and we provide a script to help you parse, validate and insert data in a PostgreSQL database. If you choose this option, consider that the DB size can be around 4 times bigger than plain files.

NSRL Lookup short template
NSRL Lookup long report

UrlScan.io

The URLScan.io analyzer has been updated with a new Scan flavour. Until now, this analyzer allowed to request report regarding a url, domain, fqdn observable. With this new flavour, anyone with a valid API key, which can be obtained for free, can request a scan on observables of the same type.

UrlScan.io short template
UrlScan.io long template

Responders

DomainToolsIris_CheckMaliciousTags

Depending on the reports generated by the DomainToolsIris analyzer, this responder adds a tag at the Case and Observable level if something malicious is found. This responder can be updated to add more custom actions depending on your needs and environment.

DomainToolsIris_AddRiskyDNSTag

Depending on on the reports generated by the DomainToolsIris analyzer, this responder adds a tag at the Case and Observable level if one of the domain observables is considered risky. This responder can be updated to add more custom actions depending on your needs and environment.

Fixes and Improvements

  • Improve TalosReputation analyzer (#521)
  • MISP WarningList analyzer fixed (#538)
  • Error fixed in ThreatCrowd (#518)
  • Encoding related bug fixed in Mailer 1_0 (#573)
  • API has changed: temporary fix for Crt_sh_Transparency_Logs_1_0 (#594)
  • Analyzers missing cortexutils in requirements (#695)
  • New mime types for Office documents in FileInfo (#705)
  • UmbrellaBlacklister analyzer now support fqdn and url observables (#547)
  • URLHaus analyzer support fqdn observables (#556)
  • Abuselpdb now support APIv2 (#618)

Get It While Supply Lasts!

If you are still using the old-style way of installing analyzers and responders, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done
for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates` menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Cerana 0.9 and Cortex 2.0.4 are Out!

We are proud to announce the immediate availability of Cerana 0.9 (TheHive 3.0.9) and Cortex 2.0.4. These hotfix releases address a number of issues and we encourage you to update your current installation at your earliest opportunity. For your comfort and sanity. Seriously.

We also took this opportunity to update Cortex analyzers to fix issues with¬†CIRCL Passive SSL, Hybrid Analysis, and the Joe Sandbox URL Analysis template. Moreover, we have updated the cortexutils library to set the taxonomy level to info if it is invalid. To upgrade¬†cortexutils‚Äč to 1.2.4:

sudo pip install -U cortexutils && sudo pip3 install -U cortexutils

To update your Cortex analyzers:

cd /path/to/Cortex-analyzers && git pull

Note: the Bluecoat analyzer was removed since it does not comply with the updated Terms of Service of Symantec Web Pulse SiteReview. Symantec does no longer permit programmatic querying of the service.

Fixes in Cerana 0.9

  • #527: display long reports when the analyst clicks on the corresponding short reports. Meh!
  • #541: make the drop-down menu for case templates scroll when there is a truckload of them.
  • #452: prevent WSAPI failure.
  • #531: fix naming inconsistencies in the Live Stream.
  • #530: correct an error when trying to analyze a filename using the Hybrid Analysis analyzer.
  • #543: generate an error if unable to contact Cortex.
  • #518: merge observable sightings when merging cases.
  • #535: fix the tag color of the PhishTank analyzer which was transparent under certain conditions.

Fixes in Cortex 2.0.4

  • #89: let a read,analyze¬†user change or display their API key.
  • #91: sort analyzers by name.
  • #92: redirect users to the index page when they click on the Cortex logo.
  • #93: under the¬†Organization > Configurations¬†page, the UI displays wrong green checkmarks for empty configurations.
  • #94: orgadmin¬†users¬†are not able to update their organization‚Äôs users after the users are created. The UI doesn‚Äôt display any error message.
  • #95: avoid ‚Äėlax programming‚Äô, Nabil styleūüėú, and strictly filter the list of analyzers in the Run dialog.
  • #90: fix Python dependency errors in docker.

Support

Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

Correction: April 14, 2018
An earlier version of this post did not mention that the Bluecoat analyzer was removed in the latest Cortex Analyzers repository release.