Hold your breath and git pull

Cortex-Analyzer 1.13.0 is out and includes new analyzers, some improvements and few bug fixes :

  • Hunter.io has been contributed by Rémi Allain (@remiallain)
  • DShield lookup contributed by Xavier Martens (@xme)
  • Pulsedive contributed by Nils Kuhnert (@3c7, TheHive Project)
  • FileInfo has been enhanced with Manalyze submodule for PE analysis ; by @lctrcl and Nils Kuhnert (@3c7)
  • Thanks to @silrumpa, the Fortiguard_URLCategory analyzer has been updated and let you configure categories and customize theirs maliciousness level
  • PhishingInitiative analyzer has been flavoured with a scan facility, thanks to Rémi Pointel

Huge thanks for the awesome work that has been performed by all our contributors!

Hunterio_DomainSearch

Hunter.io is a search engine that lets you find emails associated with a given domain name. The analyzer can be used to query the API and get a list of email addresses for a specific domain name. An API key is required to use this analyzer and can be obtained by registering on the website.

Results are displayed in TheHive in the following manner:

sc-hunterio-short

sc-hunterio-long
Hunterio_DomainSearch: short and long report samples

DShield_lookup

The analyzer lets you query the famous SANS Internet Storm Center (ISC) DShield API and look up IP address reputation. No API key is needed to run this analyzer.

Results are displayed in TheHive in the following manner:

sc-dshield-short
sc-dshield-long
DShield_lookup: short and long report templates

Pulsedive_GetIndicator

Pulsedive is a Threat Intelligence platform that allows you to enrich your observables. The analyzer can be used to query  the API and get information about a domain name,  hash, IP or URL. An API key is required to use this analyzer and can be acquired by creating an account on the webiste.

Results are displayed in TheHive in the following manner:

sc-pulsedive-short

sc-pulsedive-long
Pulsedive_GetIndicator: short and long report templates

Manalyze joins FileInfo

Manalyze is a tool developed by Ivan Kwiatkowski (@JusticeRage) that lets you analyze, operate PE (Portable Executable) and collect useful artifacts that help the analyst in determining its maliciousness.

@lctrcl wrote an analyzer that triggers Manalyze on a PE file and gives the analyst a useful report. Nils Kuhnert (@3c7) from TheHive Project then included this analyzer as a submodule into our FileInfo meta-analyzer for files ; it can run Manalyze from compiled binary, or, if your Cortex server is ready for that, through the right docker. If you decide to use the compiled binary, please follow instruction from Manalyze github page.

This submodule is disabled by default. To use it, you have to set some configuration in Cortex:

sc-fileinfo-config
FileInfo: configuration for Manalyze in Cortex

Results are displayed in TheHive in the following manner:

sc-fileinfo-short3

sc-fileinfo-long3
Manalyze submodule: short and long report templates

Fortiguard_URLCategory

This analyzer has been enhanced to let analysts choose categories considered as malicious or suspicious. It comes with a default configuration but you can setup your own by selecting the categories from the Fortiguard website.

sc-fortiguard-config
Fortiguard_URLCategory: default configuration for categories  in Cortex

Select which categories you want to be considered malicious or suspicious, and others will be considered by the analyzer as info. Analyzed observables that are not categorised by Fortigard service is considered as safe.

PhishingInitiative_Scan

PhishingInitiative has been enhanced with a new scan flavor. This let the analyst submit an URL to the webservice. An API key is needed to run this analyzer and can be obtained by registering on the website.

Results are displayed in TheHive in the following manner:

sc-phishinginitiative-short

sc-phishinginitiative-long
PhishingInitiative_Scan: short and long report samples

Bug fixes

  • #335 : fix a bug in MISP analyzer
  • #340 : fix short report template in C1fApp
  • #339 : fix short and mini reports for Domaintools Whois history flavor

Get It While Supply Lasts!

Each analyzer comes with its own, pip compatible requirements.txt file. To update your Cortex analyzers to 1.13.0, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers//requirements.txt; do sudo -H pip2 install -r $I; done && \
for I in analyzers/
/requirements.txt; do sudo -H pip3 install -r $I || true; done

Once done, do not forget to login to Cortex as an orgadmin and click on the Refresh Analyzers button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, get the latest version of  the report templates and import them into TheHive.

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Correction: September 25, 2018
The link to the report templates was incorrect.

Unveiling Synapse

When we’re not busy cooking new features, we go back to the trenches and face incidents like many of our fellow analysts who read our publications and use our tools. To do so, we swap our chef toques for firefighter helmets, not only because such shiny headwear is cool, but mainly because incident response (IR) is, at its very heart, firefighting (minus all the dangerous stuff).

If you think about it, when handling incidents you can see everything from cats in trees (spam) to major fire (APT). Thankfully, there are more cats to bring down than fire to extinguish. That being said, a big herd of cats could be a serious threat to your organization, to your mental health or both.

We tend to forget that incident handlers are humans, not robots. Unlike our metal cyberfriends, we need diversity. We can’t risk insanity like Charlie Chaplin in Modern Times if we can avoid it. Unfortunately IR can be highly repetitive, especially if you only have cats to deal with.

Some could say ‘Nah, this is minor, nothing critical here’ but at some point, an analyst brainwashed by the same tasks again and again will be led to fault. In the worst case scenario, one could see an alert and immediately categorize it as false positive without any further consideration. Because ‘this alert is always a false positive’, until the day it is not…

Automation, a Solution?

Intuitively, we look in the direction of automation in order to minimize what we call ‘zombie’ tasks: highly repetitive and brainless tasks that need to be done. We believe that doing so will allow incident handlers to focus on the analysis and not on the tedious side of IR. Ultimately, we hope it will keep analysts stimulated and in a state of alert. Also, it should reduce time and effort spent on the low-hanging fruits.

One of the most dreary tasks in our opinion is to record the context around an incident.
What is the problem? When did it happen? What’s the origin? Who are the victims? How many are there? Answers to these questions let you have an overview of what is happening and are valuable to correlate incidents. So it is worth taking some minutes to add this information to your case. Sadly, most of the time it will look like a succession of ‘Ctrl+C; Alt+Tab; Ctrl+V’ from your incident source to TheHive. Exactly the kind of tasks we want to forego.

Specifications

Having identified the threat that apathetic analysts pose, the root cause (highly repetitive tasks) and a solution (automate the recording of incident context), the question of the implementation has been raised.

The first challenge to solve is the number of incident sources. Almost everything can trigger an incident: a firewall, an IDS, antivirus, SIEM, users, etc… So the application must be designed to accept several sources and must permit to easily integrate new ones. And instead of having to configure multiple alert feeders to supply alerts to TheHive, we would have only one. To some extent, it can be assimilated to a meta feeder.

And if the application works as intended, we still have a second challenge. Let’s say you, dear reader, and ourselves use the galaxy renowned Stargazer IDS. Maybe you’d like to include the full packet capture in the case but we wouldn’t. Using the same product doesn’t mean using it the same way. So we have a variety of sources and for each source, we have a variety of configurations and workflows. Hence any app we design needs to accept multiple configurations and workflows for any given source.

Finally: the third challenge. We want to make the most out of TheHive. Creating cases, creating alerts, assigning cases, adding logs, adding observables … all those actions are not an option.

Synapse

After several trials and failures, we came up with Synapse. Basically it is a Python 3 app which sits between TheHive and your incident sources:

Screen Shot 2018-07-18 at 09.54.28.png
Synapse Overview

To solve the first and third challenge, we rely on connectors. A connector is a Python object dedicated to interact with a security device. In the picture above, you can see the Exchange Connector and TheHive Connector. To extend the number of sources, you just have to develop the connector that corresponds to your device.

Regarding the second challenge, we rely on workflows. Workflows are python scripts who use connectors to automate repetitive tasks when tracking a case. Not happy with the current workflow? Develop your own using the connectors.

At this point, you probably wonder why there’s an API in the picture above. Well, the API is the link between the user and the workflows. By hitting a specific endpoint of the Synapse’s API, the corresponding workflow will be launched. That way the user can choose what to launch, especially if they are only interested in a particular workflow. Moreover, using an API allows us to listen to TheHive’s real-time stream and initiate some actions like closing a QRadar offense when the related case is solved.

At the moment, Synapse includes the Exchange connector and the associated Ews2Case workflow. The workflow features:

  • Case creation from emails
  • Case assignment
  • Adding email bodies to task logs
  • Adding email replies to the case
  • Adding email attachments as observables

And of course, everything is done to minimize the number of clicks! Check the workflow documentation to understand how it works under the hood.

We’re still working on the QRadar connector and the associated workflows but if you can’t wait, have a look at the work done by the community like pierrebarlet’s script.

Check it Out

As usual, Synapse is an open source and free software released under the AGPL (Affero General Public License).

Synapse has its own repository. Start with the user guide and read about the workflow you want to use as you’ll need to configure it.

Troubles?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Keep Them Coming: Hashdd, URLhaus & a Revamped FileInfo analyzer

We are happy to announce the immediate availability of a new major version of Cortex-Analyzers. Version 1.11.0 includes two brand new analyzers, several updates and a few bug fixes:

  • Hashdd, contributed by  iosonogio
  • URLhaus, contributed by  ninoseki
  • The Domaintools analyzer has been updated with two new flavors: Risk and Reputation
  • The VirusTotal analyzer can (finally!) get reports for URL observables
  • MsgParser and File_Info have been merged in a new, shiny, completely rewritten FileInfo analyzer

As we are approaching the 90 analyzers mark, we wholeheartedly thank our user community for continuously contributing new analyzers, testing them and helping us improve the existing ones.

Important Notice

We made significant changes in this release in the analyzers and short reports. Prior to Cortex-Analyzers 1.11.0, the summary() function in the analyzer code generates a result such as:

"taxonomies": [{
     "level": "suspicious",
     "namespace": "DT",
     "predicate": "Risk",
     "value": "\"10\""
 }]

Double quotes were included in the resulting value. We decided to update the summary() function and make it generate the same result without double quotes :

"taxonomies": [{
     "level": "suspicious",
     "namespace": "DT",
     "predicate": "Risk",
     "value": "10"
 }]

To display the double quotes in TheHive, we updated all short reports to include them:

<span class="label" ng-repeat="t in content.taxonomies" ng-class="{'info': 'label-info', 'safe': 'label-success', 'suspicious': 'label-warning', 'malicious':'label-danger'}[t.level]">
     {{t.namespace}}:{{t.predicate}}="{{t.value}}"
</span>

Hashdd

hashdd.com is a search engine for file hashes which automatically queries 3rd party services like VirusTotal and enriches the information provided based on the 3rd party data. The analyzer includes two flavors: Status and Detail. The first one is used to query hashdd without an API key for the threat level only. The latter produces additional meta information about the sample, but requires an API key.

Results are displayed in TheHive in the following manner:

Hashdd short report

Hashdd status long report

Hashdd detail long report
hashdd — short and long report samples

URLhaus

URLhaus, a service that shares the latest malware download URLs and reports those sites to their respective hosting companies, can now be queried for domains, URLs and hashes. If the observable is found, available information will be displayed as follows:

URLhaus short report

URLhaus long report
URLhaus – short and long report samples

Domaintools Risk and Reputation

New Risk and Reputation services from Domaintools have been added as new flavors to the existing Domaintools analyzer set.

Risk Evidence

The DomainTools Risk Score predicts the risk level and likely threats from a domain that has not been observed in malicious activities, by analyzing various properties of the domain as soon as it is registered.

sc-DTrisk-short

sc-DTrisk-long
Domaintools Risk Evidence – short and long report

Reputation

The Domaintools Reputation Score gives indications about how closely a domain is related to known bad domains, actors, and IPs.

sc-DTReputation-short

sc-DTReputation-long
Domaintools Reputation – short and long report samples

An All New FileInfo

FileInfo performs local static analysis of file observables. It has been completely rewritten from the ground up to be more flexible thus it can easily be enriched with new supported file types and analysis modules. We took this opportunity to merge MsgParser, in charge of extracting and displaying Outlook emails into FileInfo.

As of this release, FileInfo now supports PDF, PE, MS Office documents and Outlook .msg files. We also added support for DDE detection and link extraction in MS Office documents, thanks to Decalage who added this in Oletools since v0.52.

sc-FileInfo-short

sc-FileInfo-long2
FileInfo – short and long report samples

Bug fixes

  • #286 : we updated the way MISP analyzer validates its SSL configuration

  • #292 : we fixed the API URL of malwares.com in the Malwares analyzer

Get It While Supply Lasts!

Each analyzer comes with its own, pip compatible requirements.txt file. To update your Cortex analyzers to 1.11.0, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip2 install -r $I; done && \
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -r $I || true; done

Once done, do not forget to login to Cortex as an orgadmin and click on the Refresh Analyzers button.

Update TheHive Report Templates

If you are using TheHive, get the latest version of  the report templates and import them into TheHive.

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Cortex4py 2 is Out!

Cortex, a free, open source software allows security analysts and threat hunters to analyze and enrich observables (IP addresses, hashes, domains, …) collected in the course of an investigation or received from third parties, for example through MISP, the de facto standard for threat sharing.

On March 29, 2018, we released Cortex 2, a major improvement over the previous version which brought, among other cool features, authentication, caching, multi-tenancy (RBAC) and rate limiting. Instead of deploying several Cortex 1 instances behind reverse proxies which would implement authentification, administrators can deploy a single Cortex 2, create multiple organizations and serve the needs of various information security populations while enjoying extra features.

On May 31, 2018, we published a brand new API guide so that developers can take advantage of the powerful REST API of the product. Sadly, Cortex4py, the FOSS Python library we provide to interact with the API was not compatible with Cortex 2. Until today.

Thanks to the hard work of our dear Nabil Adouani, we are happy to announce the immediate availability of Cortex4py 2.0.0, a complete rewrite of the library in Python 3. Cortex4py 2.0.0 is fully compatible with Cortex 2. However, it doesn’t work with Cortex 1.

While TheHive, the highly popular free and open source Security Incident Response Platform (SIRP) we develop has native support for many Cortex 2 instances, Python developers can leverage Cortex4py to interact with Cortex 2, manage organizations, users, analyzer configurations and analyze observables at scale from alternative SIRPs, SIEMs or custom scripts thanks to the 83 analyzers Cortex 2 has as of June 18, 2018.

Screen Shot 2018-06-18 at 20.01.27.png
Cortex 2: there is more than one way to interact with it

Use It

To install Cortex4py, use PIP3:

$ sudo -H pip3 install cortex4py

If you are using Python on a Windows operating system, please forgo the sudo command.

Usage

Cortex4py 2 comes with a usage guide which includes many examples. For example, if you want to fetch the last 10 successful jobs that have been executed against domain names and display the result summaries of those 10 jobs you could write something like:

Screen Shot 2018-06-18 at 19.58.45.png
Sample Python3 code to retrieve Cortex analyzer results

Migrating from Cortex4py 1

If you have already written scripts using Cortex4py 1.x (for Cortex 1), we tried to keep the already available methods. However, we recommend you adapt your code to leverage the new Cortex4py 2 classes and methods as soon as feasible. Moreover, the existing scripts must be updated to support authentication if you intend to use them with Cortex 2. Please read the Cortex4py 2 usage guide for more information.

Support

Cortex 2.0.0 is brand new software. As such, it might contain bugs and limitations. If you find any or encounter problems, please ask on our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

TheHive 3.0.10 Released

Rejoice DFIR fighters! Your favorite Security Incident Response Platform has just gotten better. TheHive 3.0.10 (codename Cerana 0.10) is just out of the oven. Thomas and Nabil, as the true code Chefs they are, have fixed a number of issues and added a few enhancements in this release. Read on, update and defend your constituency!

Fixes

  • #446: keep observable reports when performing a case merge operation
  • #592: fix the error message when an observable is created with invalid data
  • #583: cases can be merged by using their case ID
  • #584: correct error messages when performing invalid searches
  • #586: a regression was introduced that prevented analyzer names from being reflected in the modal view of mini-reports
  • #512: there were situations were mini-reports won’t display upon job success
  • #522: under certain circumstances, TheHive failed to synchronize MISP events correctly
  • #555: private dashboards must not be searchable
  • #557: fix an error when a merged case is deleted

Enhancements

  • #579: rotate application.log to keep it from becoming huge
  • #564: when submitting a job to Cortex, send the case ID as well
  • #563: poll the status of connectors (MISP, Cortex) every minute and update the UX display accordingly
  • #548: sort related cases by the amount of shared observables
  • #546: display start and end times for closed tasks

Support

Something does not work as expected? You have troubles installing or upgrading? You have spotted new bugs? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

Cortex 2.0.3 Released and Analyzer Updates

There’s a new version of your ultimate observable analysis engine in town : Cortex 2.0.3 is out!

Cortex 2.0.3 contains a few important enhancements over its predecessor and fixes a number bugs as described in the full changelog summarised below. So get it while it’s still hot out of the digital oven and let us know how tasty it is.

analyzeallthethings
Source : Quickmeme.com

Implemented Enhancements

  • #81: reflect proxy changes in the global configuration at the analyzer level
  • #82: display invalid analyzers and let orgadmins delete them
  • #85: allow orgadmins to override the default global report cache.job period per analyzer through the Web UI
  • #86: allow a job to run with arbitrary parameters

Fixed Bugs

  • #75: a version upgrade of an analyzer makes all analyzers invisible in TheHive
  • #80: fix the analyzer configuration dialog to allow orgadmins to override the auto artifact extraction at the analyzer level
  • #83: hit Nabil on the head pretty hard until the analyzer refresh UI button works (well now it does so you can stop hitting poor Nabil’s head).

Analyzer Updates

We took the opportunity of a new release to make a few updates to the public analyzers. Cortex-Analyzers 1.9.3 contains the following changes:

  • Remove the Bluecoat analyzer to comply with the new ‘no scrapping’ ToS imposed by Symantec
  • Fix the default configuration of the Cymon Check IP analyzer
  • Fix the View all VT long template
  • Make the MISP Warning Lists Analyzer ignore case sensitivity when searching for hashes
  • Restrict the Abuse Finder and FileInfo analyzer dependencies to Python 2.7

You can read the full changelog if you like but if you want to enjoy the goods right away, git pull is your friend.

Support

Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

 

Cortex 2, TheHive and a Whole Slew of Updates

After announcing Cortex 2.0.0 and TheHive 3.0.7, the first version of your favorite SIRP that is (supposedly) compatible with the brand-new version of Cortex, last week, we thought it was time to relax and enjoy the upcoming, long Easter weekend, the sunny sky of Paris (if you can pierce the veil of the Forever Grey Cloud™ that is hanging over the city of lights), and great jazz music. Heck, I even tweeted about it … only to be proven wrong by Life (and Murphy).

We literally field tested Cortex 2 for 3 weeks, we squashed bugs here and there, until almost the very last minute before the release. And yet, our QA needs to be improved by leaps and bounds as we had to release Cortex 2.0.1 one day after unveiling 2.0.0 to correct some additional bugs. And then some members of the core team and of our growing user community took it for a spin. And all hell broke lose. Well, almost 🙂

good_code
Source: XKCD

Session collisions (when TheHive and Cortex 2 are used on the same machine), analyzer malfunctions, connectivity problems … issues that were not identified during the testing phase, even in a production environment, where everything worked as expected. And we call this ‘Computer Science’. Right, right…

So we worked hard, took out our Code Hammer (it’s like Thor’s but cyber) and blasted away all the bugs that we found out or that were reported to us (arigato gozaimasu!) and we are happy to announce the immediate availability of Cortex 2.0.2, TheHive 3.0.8, Cortexutils 1.2.3 and Cortex-Analyzers 1.9.2.

TL;DR Install or upgrade Cortex 2.0.2, update Cortexutils, git pull the Cortex-analyzers repo to get the latest version of the repository, upgrade to TheHive 3.0.8, follow the Quick Start Guide and have a drink.

If you have time (which is admittedly quite scarce nowadays), please read on the changelogs:

What’s Next?

As stated in the previous post, we will release a new version of Cortex4py in order to make it compatible with Cortex 2, continue the work we started with our MISP Project friends to support MISP attribute enrichment through Cortex 2 (MISP currently only supports enrichment using Cortex 1), and perform a long-overdue overhaul of our documentation. We will also release a brand new version of TheHive4py.

Last but not least, we’ll take a hard look at ourselves and our QA. You expect us from us high quality and we hold ourselves to high standards. And we will deliver.

Support

Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.