Keep Them Coming: Hashdd, URLhaus & a Revamped FileInfo analyzer

We are happy to announce the immediate availability of a new major version of Cortex-Analyzers. Version 1.11.0 includes two brand new analyzers, several updates and a few bug fixes:

  • Hashdd, contributed by  iosonogio
  • URLhaus, contributed by  ninoseki
  • The Domaintools analyzer has been updated with two new flavors: Risk and Reputation
  • The VirusTotal analyzer can (finally!) get reports for URL observables
  • MsgParser and File_Info have been merged in a new, shiny, completely rewritten FileInfo analyzer

As we are approaching the 90 analyzers mark, we wholeheartedly thank our user community for continuously contributing new analyzers, testing them and helping us improve the existing ones.

Important Notice

We made significant changes in this release in the analyzers and short reports. Prior to Cortex-Analyzers 1.11.0, the summary() function in the analyzer code generates a result such as:

"taxonomies": [{
     "level": "suspicious",
     "namespace": "DT",
     "predicate": "Risk",
     "value": "\"10\""
 }]

Double quotes were included in the resulting value. We decided to update the summary() function and make it generate the same result without double quotes :

"taxonomies": [{
     "level": "suspicious",
     "namespace": "DT",
     "predicate": "Risk",
     "value": "10"
 }]

To display the double quotes in TheHive, we updated all short reports to include them:

<span class="label" ng-repeat="t in content.taxonomies" ng-class="{'info': 'label-info', 'safe': 'label-success', 'suspicious': 'label-warning', 'malicious':'label-danger'}[t.level]">
     {{t.namespace}}:{{t.predicate}}="{{t.value}}"
</span>

Hashdd

hashdd.com is a search engine for file hashes which automatically queries 3rd party services like VirusTotal and enriches the information provided based on the 3rd party data. The analyzer includes two flavors: Status and Detail. The first one is used to query hashdd without an API key for the threat level only. The latter produces additional meta information about the sample, but requires an API key.

Results are displayed in TheHive in the following manner:

Hashdd short report

Hashdd status long report

Hashdd detail long report
hashdd — short and long report samples

URLhaus

URLhaus, a service that shares the latest malware download URLs and reports those sites to their respective hosting companies, can now be queried for domains, URLs and hashes. If the observable is found, available information will be displayed as follows:

URLhaus short report

URLhaus long report
URLhaus – short and long report samples

Domaintools Risk and Reputation

New Risk and Reputation services from Domaintools have been added as new flavors to the existing Domaintools analyzer set.

Risk Evidence

The DomainTools Risk Score predicts the risk level and likely threats from a domain that has not been observed in malicious activities, by analyzing various properties of the domain as soon as it is registered.

sc-DTrisk-short

sc-DTrisk-long
Domaintools Risk Evidence – short and long report

Reputation

The Domaintools Reputation Score gives indications about how closely a domain is related to known bad domains, actors, and IPs.

sc-DTReputation-short

sc-DTReputation-long
Domaintools Reputation – short and long report samples

An All New FileInfo

FileInfo performs local static analysis of file observables. It has been completely rewritten from the ground up to be more flexible thus it can easily be enriched with new supported file types and analysis modules. We took this opportunity to merge MsgParser, in charge of extracting and displaying Outlook emails into FileInfo.

As of this release, FileInfo now supports PDF, PE, MS Office documents and Outlook .msg files. We also added support for DDE detection and link extraction in MS Office documents, thanks to Decalage who added this in Oletools since v0.52.

sc-FileInfo-short

sc-FileInfo-long2
FileInfo – short and long report samples

Bug fixes

  • #286 : we updated the way MISP analyzer validates its SSL configuration

  • #292 : we fixed the API URL of malwares.com in the Malwares analyzer

Get It While Supply Lasts!

Each analyzer comes with its own, pip compatible requirements.txt file. To update your Cortex analyzers to 1.11.0, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip2 install -r $I; done && \
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -r $I || true; done

Once done, do not forget to login to Cortex as an orgadmin and click on the Refresh Analyzers button.

Update TheHive Report Templates

If you are using TheHive, get the latest version of  the report templates and import them into TheHive.

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

The Mind-Boggling Implications of Multi-Tenancy

TheHive offers a powerful yet generic query API for all the data stored by the platform in the underlying Elasticsearch database.

Thanks to its DSL (Domain Specific Language), TheHive can handle complex search queries such as the following:

Among all the unassigned tasks, show me all those associated with cases which severity is high but also contain the highest number of observables which datatype is  ‘mail’

When faced with such complex queries, TheHive translates them using its DSL and sends them over to Elasticsearch to obtain the results. TheHive’s dashboards draw their power from such querties.

And while such capability is highly desirable in our opinion, a capability that we will further leverage to add a completely revamped search module in the upcoming Cerana 1 (TheHive 3.1) release, it greatly complicates RBAC (or multi-tenancy) in TheHive.

Screen Shot 2018-06-27 at 11.50.39.png
A Sneak Peek at the New Search Module of the Upcoming Cerana 1 (TheHive 3.1) Release

Indeed, in the RBAC world, the conversion of any search queries submitted to TheHive into an Elasticsearch one is fully dependent on the user context. The user view must be kept within the boundaries of the group or groups to which they belong. Each search filter,  each search parameter, must return only the results that the user can view.

The data scope needs to be clearly identified at the case level. To perform a search against task logs for example, TheHive will need to identify the parent task log, then identify the parent case and only then verify the scope. This is no small undertaking.

Similarities across cases or alerts, such as the Related Cases feature or the relationships between a given alert and existing cases, would need additional work that has not been clearly identified at this stage. But the difficulties do not stop there. Any element that has no clear relationship with case entities will have to be singled out and specific code would need to be added to limit access according to the RBAC rules. This will be clearly the case for the audit trail. Also, what should TheHive display when an analyst group is working on a case that shares observables with another one belonging to a different group? Shall it allow a limited view without any details so that groups may request from a super administrator to authorize both groups to collaborate on the investigation, something that distributed CERTs or SOCs in a large corporation may desire? Or shall it keep the data completely isolated as MSSPs which serve multiple customers with a single instance will require? We know the answer: make it configurable. But take a step back and think of the implications at the code (and security) level.

Contrary to the feature we added to Cortex 2, which allow multiple organizations to use a single Cortex instance, multi-tenancy in TheHive is a much more complex feature to implement and which is expected to have a significant impact on the platform’s performance. It will also need extreme caution to avoid blind spots that attackers (and not so innocent tenants) may exploit to circumvent scope limitations and extend their view to data they are not supposed to access. That’s why we had to delay it to Cerana 2 (TheHive 3.2), currently planned for the end of October 2018.

If you are well versed in Elasticsearch and Scala and willing to help implement this feature, please contact us at support@thehive-project.org.

 

Cortex4py 2 is Out!

Cortex, a free, open source software allows security analysts and threat hunters to analyze and enrich observables (IP addresses, hashes, domains, …) collected in the course of an investigation or received from third parties, for example through MISP, the de facto standard for threat sharing.

On March 29, 2018, we released Cortex 2, a major improvement over the previous version which brought, among other cool features, authentication, caching, multi-tenancy (RBAC) and rate limiting. Instead of deploying several Cortex 1 instances behind reverse proxies which would implement authentification, administrators can deploy a single Cortex 2, create multiple organizations and serve the needs of various information security populations while enjoying extra features.

On May 31, 2018, we published a brand new API guide so that developers can take advantage of the powerful REST API of the product. Sadly, Cortex4py, the FOSS Python library we provide to interact with the API was not compatible with Cortex 2. Until today.

Thanks to the hard work of our dear Nabil Adouani, we are happy to announce the immediate availability of Cortex4py 2.0.0, a complete rewrite of the library in Python 3. Cortex4py 2.0.0 is fully compatible with Cortex 2. However, it doesn’t work with Cortex 1.

While TheHive, the highly popular free and open source Security Incident Response Platform (SIRP) we develop has native support for many Cortex 2 instances, Python developers can leverage Cortex4py to interact with Cortex 2, manage organizations, users, analyzer configurations and analyze observables at scale from alternative SIRPs, SIEMs or custom scripts thanks to the 83 analyzers Cortex 2 has as of June 18, 2018.

Screen Shot 2018-06-18 at 20.01.27.png
Cortex 2: there is more than one way to interact with it

Use It

To install Cortex4py, use PIP3:

$ sudo -H pip3 install cortex4py

If you are using Python on a Windows operating system, please forgo the sudo command.

Usage

Cortex4py 2 comes with a usage guide which includes many examples. For example, if you want to fetch the last 10 successful jobs that have been executed against domain names and display the result summaries of those 10 jobs you could write something like:

Screen Shot 2018-06-18 at 19.58.45.png
Sample Python3 code to retrieve Cortex analyzer results

Migrating from Cortex4py 1

If you have already written scripts using Cortex4py 1.x (for Cortex 1), we tried to keep the already available methods. However, we recommend you adapt your code to leverage the new Cortex4py 2 classes and methods as soon as feasible. Moreover, the existing scripts must be updated to support authentication if you intend to use them with Cortex 2. Please read the Cortex4py 2 usage guide for more information.

Support

Cortex 2.0.0 is brand new software. As such, it might contain bugs and limitations. If you find any or encounter problems, please ask on our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

Cerana 0.9 and Cortex 2.0.4 are Out!

We are proud to announce the immediate availability of Cerana 0.9 (TheHive 3.0.9) and Cortex 2.0.4. These hotfix releases address a number of issues and we encourage you to update your current installation at your earliest opportunity. For your comfort and sanity. Seriously.

We also took this opportunity to update Cortex analyzers to fix issues with CIRCL Passive SSL, Hybrid Analysis, and the Joe Sandbox URL Analysis template. Moreover, we have updated the cortexutils library to set the taxonomy level to info if it is invalid. To upgrade cortexutils​ to 1.2.4:

sudo pip install -U cortexutils && sudo pip3 install -U cortexutils

To update your Cortex analyzers:

cd /path/to/Cortex-analyzers && git pull

Note: the Bluecoat analyzer was removed since it does not comply with the updated Terms of Service of Symantec Web Pulse SiteReview. Symantec does no longer permit programmatic querying of the service.

Fixes in Cerana 0.9

  • #527: display long reports when the analyst clicks on the corresponding short reports. Meh!
  • #541: make the drop-down menu for case templates scroll when there is a truckload of them.
  • #452: prevent WSAPI failure.
  • #531: fix naming inconsistencies in the Live Stream.
  • #530: correct an error when trying to analyze a filename using the Hybrid Analysis analyzer.
  • #543: generate an error if unable to contact Cortex.
  • #518: merge observable sightings when merging cases.
  • #535: fix the tag color of the PhishTank analyzer which was transparent under certain conditions.

Fixes in Cortex 2.0.4

  • #89: let a read,analyze user change or display their API key.
  • #91: sort analyzers by name.
  • #92: redirect users to the index page when they click on the Cortex logo.
  • #93: under the Organization > Configurations page, the UI displays wrong green checkmarks for empty configurations.
  • #94: orgadmin users are not able to update their organization’s users after the users are created. The UI doesn’t display any error message.
  • #95: avoid ‘lax programming’, Nabil style😜, and strictly filter the list of analyzers in the Run dialog.
  • #90: fix Python dependency errors in docker.

Support

Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

CorrectionApril 14, 2018
An earlier version of this post did not mention that the Bluecoat analyzer was removed in the latest Cortex Analyzers repository release.

Cortex 2.0.3 Released and Analyzer Updates

There’s a new version of your ultimate observable analysis engine in town : Cortex 2.0.3 is out!

Cortex 2.0.3 contains a few important enhancements over its predecessor and fixes a number bugs as described in the full changelog summarised below. So get it while it’s still hot out of the digital oven and let us know how tasty it is.

analyzeallthethings
Source : Quickmeme.com

Implemented Enhancements

  • #81: reflect proxy changes in the global configuration at the analyzer level
  • #82: display invalid analyzers and let orgadmins delete them
  • #85: allow orgadmins to override the default global report cache.job period per analyzer through the Web UI
  • #86: allow a job to run with arbitrary parameters

Fixed Bugs

  • #75: a version upgrade of an analyzer makes all analyzers invisible in TheHive
  • #80: fix the analyzer configuration dialog to allow orgadmins to override the auto artifact extraction at the analyzer level
  • #83: hit Nabil on the head pretty hard until the analyzer refresh UI button works (well now it does so you can stop hitting poor Nabil’s head).

Analyzer Updates

We took the opportunity of a new release to make a few updates to the public analyzers. Cortex-Analyzers 1.9.3 contains the following changes:

  • Remove the Bluecoat analyzer to comply with the new ‘no scrapping’ ToS imposed by Symantec
  • Fix the default configuration of the Cymon Check IP analyzer
  • Fix the View all VT long template
  • Make the MISP Warning Lists Analyzer ignore case sensitivity when searching for hashes
  • Restrict the Abuse Finder and FileInfo analyzer dependencies to Python 2.7

You can read the full changelog if you like but if you want to enjoy the goods right away, git pull is your friend.

Support

Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

 

Cortex 2, TheHive and a Whole Slew of Updates

After announcing Cortex 2.0.0 and TheHive 3.0.7, the first version of your favorite SIRP that is (supposedly) compatible with the brand-new version of Cortex, last week, we thought it was time to relax and enjoy the upcoming, long Easter weekend, the sunny sky of Paris (if you can pierce the veil of the Forever Grey Cloud™ that is hanging over the city of lights), and great jazz music. Heck, I even tweeted about it … only to be proven wrong by Life (and Murphy).

We literally field tested Cortex 2 for 3 weeks, we squashed bugs here and there, until almost the very last minute before the release. And yet, our QA needs to be improved by leaps and bounds as we had to release Cortex 2.0.1 one day after unveiling 2.0.0 to correct some additional bugs. And then some members of the core team and of our growing user community took it for a spin. And all hell broke lose. Well, almost 🙂

good_code
Source: XKCD

Session collisions (when TheHive and Cortex 2 are used on the same machine), analyzer malfunctions, connectivity problems … issues that were not identified during the testing phase, even in a production environment, where everything worked as expected. And we call this ‘Computer Science’. Right, right…

So we worked hard, took out our Code Hammer (it’s like Thor’s but cyber) and blasted away all the bugs that we found out or that were reported to us (arigato gozaimasu!) and we are happy to announce the immediate availability of Cortex 2.0.2, TheHive 3.0.8, Cortexutils 1.2.3 and Cortex-Analyzers 1.9.2.

TL;DR Install or upgrade Cortex 2.0.2, update Cortexutils, git pull the Cortex-analyzers repo to get the latest version of the repository, upgrade to TheHive 3.0.8, follow the Quick Start Guide and have a drink.

If you have time (which is admittedly quite scarce nowadays), please read on the changelogs:

What’s Next?

As stated in the previous post, we will release a new version of Cortex4py in order to make it compatible with Cortex 2, continue the work we started with our MISP Project friends to support MISP attribute enrichment through Cortex 2 (MISP currently only supports enrichment using Cortex 1), and perform a long-overdue overhaul of our documentation. We will also release a brand new version of TheHive4py.

Last but not least, we’ll take a hard look at ourselves and our QA. You expect us from us high quality and we hold ourselves to high standards. And we will deliver.

Support

Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

Cerana 0.4: Hotfix & Way Beyond

Update: Cerana 0.5 (TheHive 3.0.5) was released on Feb 8, 2018 to fix a regression introduced by Cerana 0.4, pertaining to the observable mini-reports. Please install 3.0.5 instead.

Waiting for the menu, one of the numerous guests of TheHive Project’s world-renowned code restaurant grows impatient. He stands up and walks toward the kitchen, looking for a waiter to chastise. It’s certainly a free meal and, looking at the reviews on CodeAdvisor, a delicious one. But guests shouldn’t wait more than two minutes to get a copy of the menu, right?

As he approaches the door leading to the kitchen, it slams open and an all-smiling, all-French-looking man dressed in a bee costume comes out carrying hot plates. Peering at their content, the once angry guest cheers up as he stares down at Cerana 0.4 or TheHive 3.0.4 if you prefer. If the looks and smells of the dish match its taste, this will be a terrific meal.

As he sees how Cerana 0.4 prompted a change of heart of the guest, the waiter coming out of the kitchen where TheHive Chefs were busy creating this new recipe goes on describing its content.

Mon bon Monsieur‘, he starts in near-perfect Parisian-English accent, ‘this is the plat du jour, or the dish of the day if you prefer. Even though the mighty Chefs insist on calling it a hotfix, not only it fixes eleven bugs in previous versions but it adds four new features and two enhancements. Would you like me to describe them?’

dt010205.gif
Source : dilbert.com © Scott Adams

The guest, feeling hungry, his mouth watery, but feeling bad about what he was about to say to the waiter just a few seconds ago nods. So the waiter goes on and describe how the latest release of TheHive will serve his hunger for efficient though free security incident response platforms even better.

Fixed Bugs

Enhancements

New Features

At this point, the guest cannot control his lust so he snatches one of the plates out the waiter’s hands, runs back to his table and start installing Cerana 0.4 to unleash its power and use it to fight cybercrime. Bon appétit !

Feeling Generous? Donate!

As you know, we are a FOSS project and donations are always welcome to make our products even better for the community.

All donations go to Creative Source, the non-profit organization we have created, and we will use them to improve TheHive, Cortex & Hippocampe but also to develop (even better) integrations with other FOSS solutions such as MISP.

So if you are feeling generous, please contact us at support@thehive-project.org.

Creative Source can also provide so-called professional, entreprise-grade support, help integrating the products, train your analysts before they drain or assist you in specific areas such as developing in-house analyzers for Cortex.

Download & Get Down to Work

If you have an existing installation of TheHive, please follow the migration guide.

If you are performing a fresh installation, read the installation guide corresponding to your needs and enjoy. Please note that you can install TheHive using an RPM or DEB package, use Docker, install it from a binary or build it from sources.

Support

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

CorrectionFeb 6, 2018
An earlier version of this post contained an incomplete sentence.