DigitalShadows Feeder 2.4.5

DigitalShadows2TH 2.4.5, a new version of the DigitalShadows feeder for TheHive, has been released with major improvements.

Source : https://www.bustle.com

Dockerise All The Things!

The most notable one is that you can now use it with docker. Run docker pull thehiveproject/ds2th and create a homedir for configuration and logs.

$ DS2TH_HOMEDIR = /opt/thehive_feeders/Digitalshadows2TH/
$ docker pull thehiveproject/ds2th:latest
$ mkdir -p $DS2TH_HOMEDIR/{config,log}
$ wget -O $DS2TH_HOMEDIR/config/config.py\
https://raw.githubusercontent.com/TheHive-Project/DigitalShadows2TH/master/config.py.template

Edit the config.py before runing the docker command below:

$ docker run --rm --net=host --mount\
type=bind,source="$DS2TH_HOMEDIR"/config,target=/app/config --mount type=bind,source="$DS2TH_HOMEDIR"/log,target=/app/log certbdf/ds2th <OPTIONS>
Quick Installation

If that sounds still complicated, worry not! We also wrote a little program that helps you prepare the environment and install everything. We ensured that it works well on Ubuntu 18.04. The program uses two environment variables to set up everything: FEEDERS_SYSACCOUNT and FEEDERS_HOMEDIR :

$ export FEEDERS_SYSACCOUNT=thehive
$ export FEEDERS_HOMEDIR=/opt/thehive_feeders

There are also sane, default settings in case you did not set any value. DigitalShadows2TH’s home directory will be set to /opt/thehive_feeders/DigitalShadows2TH. To use the script, run the following command line and follow the instructions:

 $ wget -qO- https://raw.githubusercontent.com/TheHive-Project/DigitalShadows2TH/mater/INSTALL/install_with_docker.sh | sudo -E bash -

Custom Case Templates

Previous versions of DigitalShadows2TH allowed only one case template to be associated with alerts created by the feeder in TheHive. Starting from DigitalShadows2TH 2.4.0, you can define a case template for each type of incidents raised by DigitalShadows in the configuration file.

The configuration pertaining to TheHive looks as follows:

TheHive = {
 'proxies':{
     'http': '',
     'https': ''
},
   'url':'THEHIVE_URL',
   'key':'THEHIVE_API_KEY',
   'templates': {
       'default':''
  }
}

Each incident type in DigitalShadows can be associated with a case template in TheHive, for example:

TheHive = {
  'proxies':{
      'http': '',
      'https': ''
  },
    'url':'THEHIVE_URL',
    'key':'THEHIVE_API_KEY',
    'templates': {
        'default':'MY_DEFAULT_CASE_TEMPLATE_FOR_DIGITALSHADOWS_INCIDENTS',
        'DATA_LEAKAGE': 'MY_DATA_LEAKAGE_CASE_TEMPLATE',
        'CYBER_THREAT': 'MY_CYBER_THREAT_CASE_TEMPLATE'
    }
}

A template can be defined for all the following DigitalShadows incident types:

  • DATA_LEAKAGE
  • CYBER_THREAT
  • PHYSICAL_SECURITY
  • SOCIAL_MEDIA_COMPLIANCE
  • BRAND_PROTECTION
  • INFRASTRUCTURE

A default template can be defined for DigitalShadows incidents. If no template is found for a specific incident type, the feeder looks for the default template. if no default template is found, an empty case will be created by when importing the alert.

Update or Install

If you are not using docker, just pull the repository and update your configuration file with the new templates part for TheHive.

Update your Repository
$ cd /opt/TheHive_feeders/DigitalShadows2TH/
$ git pull

The configuration file has changed, so you need to update yours before running the program. A new templates section has been added for TheHive and the path has changed. It is now in the config/ directory of the project.

Install and Use via the Code Repository
$ cd /opt/TheHive_feeders 
$ git clone https://github.com/TheHive-Project/DigitalShadows2TH.git

After that, follow the prerequisites and edit the configuration file. In /opt/TheHive_feeders/DigitalShadows2TH/config/ copy config.py.template to config.py and modify it.

Use cases and detailed configuration instructions can be found in the README file in the repository.

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

TheHive 3.3-RC2, Hot out of the Oven

TheHive Project’s code Chefs, dressed in their outright haute cuisine outfit, including the traditional toque blanche, have been quite busy lately, working on dockerizing all the Cortex analyzers (more on this later in an upcoming post), and doing tedious work to prepare the replacement of Elasticsearch by a GraphDB which will help us finally release much-awaited features such as multi-tenancy, delayed for way too many months (yeah, yeah, don’t chastise them but feel free to help them). In the meantime, they found enough bandwidth to release a new major version of TheHive.

Version 3.3, currently a release candidate includes several bug fixes and many new features as outlined below. Please note that TheHive 3.3-RC2 is beta software. As all our other release candidates, you can grab it from the pre-release, beta repositories. As usual, we would truly appreciate your help making it a great stable release by testing it as thoroughly as possible and reporting back any bugs or issues you encounter so we can address them before the final release.

Check TheHive Installation guide for further details.

Wait! Where’s RC1?

TheHive 3.3-RC1 was very short-lived. Few hours after its release, and thanks to Chris (a.k.a. crackytsi on GitHub), Thomas Franco, our back-end mastermind, discovered an issue with the Debian 8 and Debian 9 packages.

New Features

  • #836: add a new exportCaseTags parameter to the MISP configuration section. If set to true, all the tags associated with a case will be exported along with it to MISP.
  • #861: add support for Java higher than 8, such as OpenJDK 11.
  • #271: bulk merge alerts into a case. Select multiple alerts at once and create a single case out of them or merge them into an existing case using its ID.
  • #824: add ability to sort alerts by reference, status, type, source…
  • #826: when previewing an alert, there are sometimes no overlap with an existing case. However, an analyst might already know, thanks to HI (Human Intelligence), that the alert should be merged into a specific case. This is now possible thanks to a new button.
  • #769: improve case template selection for case creation. If you have defined a large set of case templates, you will be able to sort/filter to find the case you want to use when creating a New Case.
New case template selector
  • #657: add observable tags auto-completion. Contributed by Tyler Chong (Thanks!).
Observable tag auto-completion

Fixed Bugs

  • #864: do not return a session cookie when making an API call.
  • #856: there was a bug where after a followed alert PATCH, if the alert has already been promoted to a case, the case is not updated. Now, if the alert has follow=true, if it gets updated, its status is set to Updated and the related case is updated too.
  • #845: assigned but unstarted tasks were not showing up in My Tasks.
  • #844: enable user account locking through the Delete API endpoint.

Stuck?

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

TheHive4py 1.5.1 Released

When you need to interact with TheHive’s REST API and you ain’t shy of working with Python, TheHive4py is the way to go. It’s a free, open source library we provide to allow you to easily create alert feeders, automate certain tasks like creating cases, assign them to analysts and much more. For example, Synapse, DigitalShadows2TH and Zerofox2TH leverage the library to send alerts to your favourite SIRP/SOAR.

Sometime ago, we decided that it was time to overhaul the whole library and we began working on version 2.0.0 which will be easier to use. It should also support the full set of TheHive’s REST API calls. In the meantime we decided to release version 1.5.0, shortly followed by version 1.5.1 to support some new functionality contributed by our user community and correct a few issues.

code_quality
Source : XKCD

New Features Introduced in 1.5.0

Bugfixes Introduced in 1.5.0

  • #80: Prevent max recursion depth exceeded error, contributed by Psynbiotik

New Features Introduced in 1.5.1

Important note: TheHive4py 1.5.1 does not work with TheHive 3.0.10 or earlier versions. Please stick with 1.5.0 if you are using those versions.

Updating/Installing

To update your existing package:

$ sudo pip install thehive4py --upgrade

If you are just getting started with TheHive4py, you can forgo the --upgrade at the end of the command above.

But I just Wanna Play!

If you’d like to play around with TheHive4py 1.5.1, TheHive 3.1.1., Cortex4py 2.0.1 and Cortex 2.1.1, please download the training VM.

Paris? Are you There?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. As usual, we’ll be more than happy to help!

Correction: October 12, 2018
As reported by Robin Hahling, TheHive 1.5.1 does not work with TheHive 3.0.10 or earlier versions.

EmlParser: a New Cortex Analyzer for EML Files

The new EmlParser analyzer which we included in Cortex-Analyzers 1.12.0 leverages the eml_parser python library written by GOVCERT-LU. It parses EML email,  a MIME RFC 822 standard format, and extract all the information to help the analyst triage and investigate. EmlParser will prove very useful when analyzing observables imported from Synapse alerts.

You might notice that the analyzer’s requirements.txt installs the eml_parser library from one of our repositories. The original library dependencies contains file_magic library which brokes other analyzers that use python-magic. GOVCERT-LU is addressing this situation in their code but the installation process still considers file-magic as a mandatory library. We decided to consider it as an extra requirement.

Screen Shot 2018-07-26 at 08.19.11.png

Screen Shot 2018-07-26 at 08.19.31.png
EmlParser: short and long report samples

Get It While Supply Lasts!

To update your Cortex analyzers to 1.12.0, run the following commands:

cd path/to/Cortex-Analyzers

git pull

for I in analyzers/*/requirements.txt; do sudo -H pip2 install -r $I; done && \

for I in analyzers/*/requirements.txt; do sudo -H pip3 install -r $I || true; done

Once done, do not forget to login to Cortex as an orgadmin and click on the Refresh Analyzers button.

Update TheHive Report Templates

If you are using TheHive, get the latest version of  the report templates and import them into TheHive.

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

Cerana 0.9 and Cortex 2.0.4 are Out!

We are proud to announce the immediate availability of Cerana 0.9 (TheHive 3.0.9) and Cortex 2.0.4. These hotfix releases address a number of issues and we encourage you to update your current installation at your earliest opportunity. For your comfort and sanity. Seriously.

We also took this opportunity to update Cortex analyzers to fix issues with CIRCL Passive SSL, Hybrid Analysis, and the Joe Sandbox URL Analysis template. Moreover, we have updated the cortexutils library to set the taxonomy level to info if it is invalid. To upgrade cortexutils​ to 1.2.4:

sudo pip install -U cortexutils && sudo pip3 install -U cortexutils

To update your Cortex analyzers:

cd /path/to/Cortex-analyzers && git pull

Note: the Bluecoat analyzer was removed since it does not comply with the updated Terms of Service of Symantec Web Pulse SiteReview. Symantec does no longer permit programmatic querying of the service.

Fixes in Cerana 0.9

  • #527: display long reports when the analyst clicks on the corresponding short reports. Meh!
  • #541: make the drop-down menu for case templates scroll when there is a truckload of them.
  • #452: prevent WSAPI failure.
  • #531: fix naming inconsistencies in the Live Stream.
  • #530: correct an error when trying to analyze a filename using the Hybrid Analysis analyzer.
  • #543: generate an error if unable to contact Cortex.
  • #518: merge observable sightings when merging cases.
  • #535: fix the tag color of the PhishTank analyzer which was transparent under certain conditions.

Fixes in Cortex 2.0.4

  • #89: let a read,analyze user change or display their API key.
  • #91: sort analyzers by name.
  • #92: redirect users to the index page when they click on the Cortex logo.
  • #93: under the Organization > Configurations page, the UI displays wrong green checkmarks for empty configurations.
  • #94: orgadmin users are not able to update their organization’s users after the users are created. The UI doesn’t display any error message.
  • #95: avoid ‘lax programming’, Nabil style😜, and strictly filter the list of analyzers in the Run dialog.
  • #90: fix Python dependency errors in docker.

Support

Something does not work as expected? You have troubles installing or upgrading? Spotted new bugs? No worries, please open issues on GitHub or comment on existing ones, join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

CorrectionApril 14, 2018
An earlier version of this post did not mention that the Bluecoat analyzer was removed in the latest Cortex Analyzers repository release.

TheHive4py 1.4.0 Released

Version 1.4.0 of the Python API client for TheHive is now available. It is compatible with the freshly released Cerana (TheHive 3.0.0).

We’d like to thank Nick Pratley, a frequent contributor, Bill Murrin, Alexander Gödeke and “srilumpa” for their code additions and documentation.

To update your existing package:

$ sudo pip install thehive4py --upgrade

If you are just getting started with TheHive4py, you can forgo the --upgrade at the end of the command above.

New Features

  • #5: Add a method to update a case, contributed by Nick Pratley
  • #34: Add a get_task_logs method in order to obtain all the task logs associated with a given taskId. Contributed by Bill Murrin
  • #37: A new, very cool case helper class by Nick Pratley
  • #39: Add support for custom fields to the case model
  • #40: Ability to run a Cortex analyzer through the API by Alexander Gödeke
  • #45: Simplify case creation when using a template by providing just its name
  • #49: Add a query builder capability to support TheHive’s DSL query syntax

Paris? Are you There?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. As usual, we’ll be more than happy to help!

VMRay, FireHOL, Joe Sandbox & Fortiguard Analyzers

We are pleased to announce the availability of 2 new Cortex analyzers and an update to 2 existing ones:

  • New: VMRay and FireHOL
  • Updated: Joe Sandbox and Fortiguard URL Category

We would like to thank Nils Kuhnert from CERT-BUND, CERT-BDF and Eric Capuano for their precious contributions.

To install the new analyzers, grab the Cortex-Analyzers repository and unpack its content (or git pull the master  branch) in your existing /path/to/cortex-analyzers. Then follow the Cortex analyzers guide.

To import the new report templates in your instance of TheHive:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu
  • click on Import templates button and select the downloaded package

VMRay

The VMRay analyzer has been submitted by Nils Kunhert from CERT-BUND. It lets you run a file in a local or remote (cloud) VMRay sandbox. The analyzer also lets you check existing analysis reports.

The analyzer accepts files and hashes as input. VMRay is a commercial service and you need an API key to run the analyzer. To make it work, install the requests Python library. It should already have been installed since it is used by other analyzers as well.

To use the analyzer, add the following section to the Cortex configuration file (application.conf):

VMRay {
 url = ""
 key = ""
 certpath = ""
 }

When called from TheHive, the following output is produced:

sc-short-vmray.png

sc-long-vmray.png
TheHive: VMRay Analyzer – Short and Long Report Samples

Important note: an analysis on VMRay, like on any other sandbox, can take a long time. That is why the analyzer tries to fetch the report until it is ready.

FireHOL

The FireHOL analyzer has been submitted by Nils Kuhnert from CERT-BUND. It lets you use the lists maintained by FireHOL project and check if an IP resides in one of them. FireHOL is an open source project. The analyzer reports the block lists in which an IP resides with the latest updated ones displayed first. To make it work, you’ll need to download the lists in a directory first (and it would be wise to do it on a regular fashion using a cron entry for example):

git clone https://github.com/firehol/blocklist-ipsets

The FireHOL analyzer depends on the following librairies:

ipaddress
pytz
dateutil
datetime

Add the following section to the Cortex configuration file (application.conf) to activate the analyzer:

FireHOLBlocklists {
 blocklistpath = ""
 ignoreolderthandays = <int>
 }

The ignoreolderthandays parameter lets you tell the analyzer to ignore matches found in lists that have not been refreshed in <int> days where <int> is an integer.

When called from TheHive, the following output is produced:

sc-short-fireHOL.png

sc-long-firehol.png

TheHive: FireHOL Analyzer – Short and Long Report Samples

Joe Sandbox

Thanks to CERT-BDF, the Joe Sandbox analyzer has been updated to support Joe Sandbox Cloud service beside the on-premises version (Ultimate). Like with other Joe Sandbox services, you need to add the following section to the Cortex configuration file (application.conf):

JoeSandbox {
 url = ""
 apikey = ""
 }

Fortiguard URL Category

Thanks to Eric Capuano, the Fortiguard URL Category analyzer is working again. Eric has modified it to handle the changes made by Fortiguard to their free online API.

Correction: May 23, 2017
An earlier version of this post used ignoredays instead of ignoreolderthandays for the FireHOL Blocklists analyzer. This parameter has also been described.