TheHive 4.0.4 and TheHive4py 1.8.1: alerts got more APIs

Dear community, the new year has brought us another opportunity to build new features in your favorite Security Incident Response Platform, TheHive. We wish you a cheerful new year ahead and we thank you for being beside us all these years.

Last week, we released TheHive 4.0.4 and TheHive4py 1.8.1, and here is the official announcement including the details of the new features.

These releases focused on adding more capabilities to play with alert observables and give more flexibility when building alert feeders.

Please find the change logs for more details:

What’s new in TheHive

New Alert observable APIs

The major change in TheHive 4.0.4 is related to alert management. In TheHive 3, alert observables were included in the alert as an array of observable objects, and not as independent objects with links to the alert itself. This data model made alert observables CRUD operations, a bit challenging.

TheHive 4 has a better design for this, and alert observables have their own existence, and can be added/updated and deleted independently from the alert object.

This new design allows adding dedicated API endpoints to:

  • Add an observable to an existing alert;
  • Update the data of an existing alert observable;
  • Delete an observable from an alert.

Those APIs are not used by the user interface for now.

New Alert properties

This release introduced a new property called `importDate`. It represents the date at which an alert has been merged into a new/existing case. This property is then used to:

  • Allow filtering the alert list, for example: “List the alerts merged today”
  • Display the duration between the alert creation and its merge into a case.
Alert list showcasing alert importDate

This new property is of course available on the dashboard creation UI as a date field, among others:

  • imported: true if the alert has been merged
  • `handlingDurationInSeconds`: number of seconds before importing an alert
  • `handlingDurationInMinutes`: number of minutes before importing an alert
  • `handlingDurationInHours`: number of hours before importing an alert
  • `handlingDurationInDays`: number of days before importing an alert

To showcase the mentioned new properties, here are some screenshots:

importDate field used on a line chart as date field for x-axis
Simple line chart using the imported filter
Dashboard including two charts using the newly introduced alert fields

What’s new in TheHive4py

The 1.8.1 release of TheHive4py mainly focuses on adding support to the new alert APIs introduced by TheHive 4.0.4. It comes with 3 new functions:

  • `create_alert_artifact` to allow developers adding a new artifact to an existing alert
from thehive4py.api import TheHiveApi
from thehive4py.models import Tlp

THEHIVE_URL = 'http://127.0.0.1:9000'
THEHIVE_API_KEY = '**YOUR_API_KEY**'

api = TheHiveApi(THEHIVE_URL, THEHIVE_API_KEY)

# Instanciate a new domain artifact
artifact = AlertArtifact(dataType='domain', data='malicious-domain.tld', ignoreSimilarity=True, ioc=True)
api.create_alert_artifact(ALERT_ID, artifact)

# Instanciate a new file artifact
artifact = AlertArtifact(
    dataType='file', 
    data='malicious-file.exe', 
    ignoreSimilarity=False, 
    ioc=True, 
    sighted=True, 
    tlp=Tlp.RED.value)
api.create_alert_artifact(alert_id, artifact)
  • `update-alert-artifact` to allow updating the data of an existing alert artifact:
from thehive4py.api import TheHiveApi
from thehive4py.models import Tlp

THEHIVE_URL = 'http://127.0.0.1:9000'
THEHIVE_API_KEY = '**YOUR_API_KEY**'

api = TheHiveApi(THEHIVE_URL, THEHIVE_API_KEY)

# Create a new domain artifact
artifact = AlertArtifact(dataType='domain', data='malicious-domain.tld', ignoreSimilarity=True, ioc=True)
response = api.create_alert_artifact(ALERT_ID, artifact)

# Update its tlp, sighted and ignoreSimilarity flags
artifact_data = response.json()[0]
artifact_data['tlp'] = Tlp.RED.value
artifact_data['sighted'] = True
artifact_data['ignoreSimilarity'] = False

new_artifact = AlertArtifact(json=artifact_data)
api.update_alert_artifact(artifact_data['id'], new_artifact, fields=['tlp', 'ioc', 'ignoreSimilarity'])
  • `delete_alert_artifact` to allow removing an existing artifact from an existing alert
from thehive4py.api import TheHiveApi

THEHIVE_URL = 'http://127.0.0.1:9000'
THEHIVE_API_KEY = '**YOUR_API_KEY**'

api = TheHiveApi(THEHIVE_URL, THEHIVE_API_KEY)

# Delete alert artifact
api.delete_alert_artifact(ARTIFACT_ID)

Note that these new three methods are only available when using TheHive4py with TheHive 4.0.4+

You can find more details on the official documentation of TheHive4py.

Updating/Installing

To update your existing package to version 1.8.1:

$ sudo pip install thehive4py --upgrade

How to report issues

Please open an issue on GitHub with the dedicated template for TheHive 4 or the dedicated form for TheHive4py. We will monitor them closely and respond accordingly. 

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Discord, or send us an email at support@thehive-project.org. We will be more than happy to help!

Xmas Release: TheHive 4.0.3

This year is coming to an end. It’s a new opportunity to our team to release a new version of your favorite free and open source incident response platform, TheHive. We don’t aim to bother you during your Xmas holidays, but we thought of those who will be in front of their screens during the years’ last days.

Today, we announce the release of 4.0.3 that comes with some interesting enhancements and bug fixes.

What’s new?

The major features added in this release are:

  • improvement of the task management capabilities for collaboration, by adding a new flag called “Action Required”
  • review and improvement of the implementation of MISP synchronisation filters
  • new API capability to allow searching for Alerts per observables
  • new S3 provider and configuration for file storage (observables and task log attachments)

Require an action on a task

This feature is useful for collaboration to allow a master team requiring actions from the teams it collaborates with. Let’s explain it using a concrete example. Imagine a TheHive instance defining two organisations: SOC and CERT, both of them working on a Case led by the CERT team who is the owner of the Case.

For one of the tasks, the incident handler from the CERT team needs an action from the SOC team on a Task called “Gather evidences or IOC”. In the Task’s details page, at the “Sharing” section, for each “Share”, a new button “Require Action” is displayed. It allows any user with `manageTask` permission to enable the flag for the specific organisation for that specific Task.

The incident handler can also require an action from his team members.

Task details, Require Action from its own team
Task details, Require Action from another team

When the user clicks the “Require Action” button, TheHive asks if a Task Log needs to be added to the Task, to explain the required action. The user can proceed without adding a Task Log

Require Action confirmation dialog

If the user clicks on “Yes, add log”, a dialog is displayed asking the user to type a Task Log message and optionally add an attachment:

Add Task Log dialog

Once confirmed, the action is completed and the Task is marked as requiring action from the right team.

Action required from the SOC team on the “Gather evidences or IOC” Task

This feature comes not only with these Task related buttons but also includes some features for Case and Task listing pages, including filtering: its easier to list the cases where at least one task is still requiring an action from a team:

List of cases requiring actions

When navigating to the Task list, the user can easily see which Task needs an action:

Task list, filtered by the action required flag

When a user accesses a Task that is marked as requiring an action, the Task details page displays a warning message, with a “Mark as Done” button, that when clicked, confirms the completion of the required action:

Action Required warning message

When clicked, the “Mark as Done” button goes through the same confirm dialog as described above, allowing to optionally add a Task Log.

Review MISP filters configuration

TheHive 4.0.3 has reviewed and improved the performance of the MISP synchronisation services and added a new config to whitelist events by organisation, not only by tags. Below is a non exhaustive configuration of a MISP server, where the `whitelist.organisation` filter is used:

play.modules.enabled += org.thp.thehive.connector.misp.MispModule
misp {
  interval: 1 hour
  servers: [
    {
      name = "local"            # MISP name
      url = "http://localhost/" # URL or MISP
      auth {
        type = key
        key = "***"             # MISP API key
      }

      ...

      # Organization and tags 
      whitelist {
        organisation = ["good organisation"]
      #  tags = ["tag1", "tag2"]
      }
    }
  ]
}

Search for alerts by observable conditions

In TheHive 3, Alert observables are stored within the Alert data, not as un independent piece of data, and querying them is not possible through the `_search` APIs and the Query DSL.

In 4.0.3 this limitation has been removed, allowing querying an alert with conditions on it’s child observables.

Using TheHive4py, the following code is now possible:

import json
from thehive4py.api import TheHiveApi
from thehive4py.models import *
from thehive4py.query import *

THEHIVE_URL = 'http://127.0.0.1:9000'
THEHIVE_API_KEY = 'API_KEY'

api = TheHiveApi(THEHIVE_URL, THEHIVE_API_KEY)

# Define the query
query = And(
    Eq('source', 'THEHIVE-PROJECT'),
    Eq('severity', Severity.MEDIUM.value),
    Child('alert_artifact', And(
        Eq('dataType', 'hash'),
        Eq('data', 'A_HASH_VALUE')
    )),
    Like('title', '*MALSPAM*')
)

# Search for alerts
response = api.find_alerts(query=query, range='all')

# Print 
print(response.json(), indent=4)

The code snippet above, searches for Alerts of with:

  • source='TheHive-Project'
  • severity=medium
  • title including the word “MALSPAM”
  • having a specific `hash` observable

S3 storage support

File storage is used in TheHive to store attachments. TheHive now supports a new type of storage in addition to hadoop and local file system: Amazon S3.

S3 storage provider can be configured by specifying endpoint, region and credentials. The settings must be located in the provider section, in application.conf:

storage {
  provider: s3
  s3 {
    bucket = "thehive"
    readTimeout = 1 minute
    writeTimeout = 1 minute
    chunkSize = 1 MB
    endpoint = "http://s3.amazonaws.com"
    accessKey = "xxx"
    secretKey = "xxx"
    region = "us-west-1"
  }
}

How to install/update ?

Our installation guides are up-to-date.

Important note

If you are updating an existing instance of TheHive 4.0.0, ensure to read the installation guide. Since our recent release of TheHive 3.5.0, we updated our repositories. The codename of repositories containing all new released packages is now named release, instead of stable. So update your apt or yum repository file.

How to report issues

Please open an issue on GitHub with the dedicated template for TheHive 4. We will monitor them closely and respond accordingly. 

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Discord, or send us an email at support@thehive-project.org. We will be more than happy to help!

It’s not Patch Friday… TheHive 4.0.2 released

Last Friday, our team released a significant number of changes and fixes, improving TheHive 4. The community was very reactive and hurried to test it. Today we are announcing a patch release to fix annoying issues we decided to quickly patch: welcome to 4.0.2.

The major issue is related to Alert bulk merging which is part of the big refactoring of the Alert listing section.

Well, s**t happens, but as a gentlemen, we owe you a feature, or two 🙂

What’s new?

In addition to the Alert bulk merging issue, we fixed bugs related to:

  • Migration
  • Filtering observables by attachement attributes
  • Backward compatible APIs

Curious to read the detailed change log? Here your go.

TheHiveFS, do you remember?

It stands for TheHive File System, a feature we released with TheHive 4.0-RC1. In today’s milestone we are improving the security of this feature by adding a new user permission, required to have access to TheHiveFS features.

The newly created accessTheHiveFS permission is included by default in both org-admin and analyst user profiles.

Refined editable fields, for a better UX

In TheHive UI, editing objects relies on editable fields instead of dedicated form to edit objects. This means you can update a case title by clicking the Blue Pencil icon displayed when you mouse-over the case title for exemple.

In this release we refactored all the editable fields to provide a better user experience:

  • You no longer need to click just on the `Blue Pencil` icon to switch to edit mode, you just need to click on any value, for example the Case assignee field, or the Observable tags field:
Editable fields on mouse over events

Editable fields have now a `Clear` option, allowing users to unset the value of an attribute:

Editable custom fields can be set empty

This new improvement benefits to the custom fields sections in Case and Alert details sections, as showcased above.

Configurable layout of custom fields

In older TheHive versions, custom fields were displayed using a single column. Cases with big number of custom fields produced a long scrolling Case details pages, so we decided in TheHive 4 to use a 3-column layout to reduce the resulting scroll fatigue.

Now some users are complaining because of long custom fields values not being correctly displayed. So we decided to let users choose their preferred layout.

1-column layout to display custom fields in a Case details page
2-columns layout

How to install/update ?

Our installation guides are up-to-date.

Important note

If you are updating an existing instance of TheHive 4.0.0, ensure to read the installation guide. Since our recent release of TheHive 3.5.0, we updated our repositories. The codename of repositories containing all new released packages is now named release, instead of stable. So update your apt or yum repository file.

Have you got a minute to let us know how you use TheHive ?

We would love to hear from you (10 questions < 1minute) : https://t.co/l5osjvQ1k7

Many thanks for taking the time 🙏

How to report issues

Please open an issue on GitHub with the dedicated template for TheHive 4. We will monitor them closely and respond accordingly. 

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Discord, or send us an email at support@thehive-project.org. We will be more than happy to help!

It’s now time for TheHive 4 to get an update: TheHive 4.0.1 is out!

Back in July, we announced TheHive 4.0, the foundation for future releases of your favorite Open Source SIRP.

Today, we are pleased to deliver one of the biggest patch releases we’ve done so far, in terms of number of fixed issues: TheHive 4.0.1! Not only did we squash lots of bugs, we also included some cool improvements we didn’t want to hold until the next version.

As for TheHive 3 and Cortex unveiled few days ago, we fixed the vulnerabilities reported by Play framework in October as well.

A huge work has been accomplished to fix a backlog of bugs (thanks to the community) in several parts of the product:

  • Configuration;
  • API;
  • Dashboards;
  • Livestream;
  • Cases;
  • Alerts;
  • Observables;
  • Multi Factor authentication;
  • Active Directory authentication;
  • Export to MISP.

You can find details of the 80+ issues included in this milestone: Changelog file.

Wait, there is also new stuff:

  • Cases and Alerts similarity fine tuning;
  • Similar cases in Alert preview panel refined;
  • Filtering capabilities improvements;
  • New UI settings & options;
  • Custom fields in Case and Alert lists;
  • Statistics panel improvements;
  • API key in user settings page;
  • More migration tool options.

Cases and Alerts similarity fine tuning

In TheHive, correlation between Alerts and Cases, and between Cases themselves, is computed throughout their respective Observables:

  • A Case is considered similar to an Alert, if they have at least one common observable;
  • A Case is considered related to another Case, if they have at least one common observable.

In some situations, those relationships are undesirable when based on useless/meaningless observables.

Starting with TheHive 4.0.1, Observables are enriched with a new attribute called ignoreSimilarity, set to false by default. Now, when this attribute is set to true, an observable is simply not taken into account in the relationship between two cases, and is not involved in the similarity calculation, for Cases and Alerts.

This attribute is displayed alongside with the IOC and Sighted flags, in the Observables list – which has been tweaked for the occasion – as well as in the detailed view.

IgnoreSimilarity flag in the revamped Observables list

And you can also edit this for a set of selected observables.

Edit “Ignored for similarity”

Long story short, if you want to break a correlation, you just need to set the linking Observable‘s ignoreSimilarity flag to true.

Alerts and similar Cases

The attribute mentioned earlier is taken into account when correlating Alerts with existing Cases : the Similar Cases view in the Alert preview dialog.

Observables with ignoreSimilarity set to true will be ignored when fetching cases similar to the current Alert, whether it is in the Alert or the Case side.

A forthcoming update of thehive4py library will allow to set this attribute in Alerts, along with many other new things.

The Similar Cases view has been thoroughly enhanced to provide more highlights facilitating alert triage and the merging of alerts into existing cases. First, a column named “Matched” has been added. It contains the list of observable dataTypes and the number of occurrences matching the Alert : this should be very useful to identify quickly what type of data has matched the alert content. A filter on this column has also been added.

And there are more refining capabilities in this view:

  • Sorting and Filtering on Title of Similar Cases;
  • Specifying a minimum observables rate that matches the Alert;
  • Specifying a minimum IOCs rate that matches the Alert.
Refining the Similar Cases view during Alerts triage

… And, above all, columns can now be sorted. With all these new capabilities, triage and merge Alerts in Cases should be easier.

Filtering capabilities improvement

In TheHive 4, filter forms have the same structure all over the product. Any improvement in the area, benefits to all the listing pages.

For example, in the alert similar cases tab, a new filter form has been included. You can quickly select the type of Cases you want to display, but also use the advanced rules and search criterias.

In this release, the filter component has new capabilities:

  • Date filter: now allows setting dynamic ranges instead of setting from and to fixed date values. Possible values are:
    • Today;
    • Last 7 days;
    • Last 30 days;
    • Last 3 months;
    • Last 6 months;
    • Last year;
    • Custom: allowing the old-fashioned way of setting date ranges.
  • Tag filter: Autocomplete is now enabled.
Advanced filtering for Similar Cases during Alert triage
Autocompleted tags when adding a filter

In TheHive 4.0.1, there is one more place that welcomed a filter form: the list of Alert observables in the preview dialog.

New Org level UI settings

UI configuration page is the place where TheHive adds new options to customise the behavior of the UI, for a given organisation. This is an area that will be expanded over time.

For this milestone, two new options have been introduced:

  • Allow merging Alerts in resolved cases;
  • Default filter of Alert similar cases.
Default filter for Similar Cases in Alerts preview

The first one is self-explanatory and will hide the `Merge into case` button in the Alert similar cases section

The second one allows an administrator define, for an organisation, the default filter to apply to Alert similar cases view. For example: “show only potential similar cases created in the last 7 days and are still open”.

If during the triage, the filter has been updated by an analyst, the default filter cas be retrieved through Default filter option of the Quick Filters.

Retrieve default filter in alert similar cases

Custom fields in Case and Alert lists

“Damn, why I’m not able to see my custom fields in my cases list”, “It’s a shame I cannot filter my alerts with my custom defined properties”… We are pretty sure you said it, at least once.

Well, you dreamt of it, and here we go, we did it for Cases and Alerts

  • You will be able to see all the custom fields you defined in your Case Templates and propagated to your Cases;
  • Custom fields with undefined values are not displayed;
  • Custom fields are shown with respect of the order you define;
  • Clicking on a custom field, filters your lists using the selected value;
  • You can enable displaying the custom fields using the toggle button on the toolbar located on top of the filter form;
  • All those capabilities are available in Cases and Alerts views.
Custom Fields in Alerts list

New statistics view in Cases and Observables lists

The Stats view has been revamped to represent the statistics using both tables and shiny colourful donuts (not only for managers).

This improvement has been made in Cases, Observables and Alerts views.

New “Stats view” in Cases list

User settings

This feature has been initially introduced in Cortex and was missing in TheHive. All users can now access their API key, copy or update it, as long as an administrator created it first. No more API keys sent over emails or chat applications.

User settings with API key enabled

Migration

The migration assistant program has been updated with a few bug fixes and new filtering options. When migrating, you can now specify:

  • If you want to include or exclude Alerts generated from specific Sources or by their Type;
  • If you want to include Audit trails regarding specific objectType (case, case_task, case_observable …);
  • If you want to filter out Audit trails with specific actions (Update, Creation, Delete).
/opt/thehive/bin/migrate --help

[..]  
  --include-alert-types ,...
                           migrate only alerts with this types
  --exclude-alert-types ,...
                           don't migrate alerts with this types
  --include-alert-sources ,...
                           migrate only alerts with this sources
  --exclude-alert-sources ,...
                           don't migrate alerts with this sources
[..]   
  --include-audit-actions 
                           migration only audits with this action (Update, Creation, Delete)
  --exclude-audit-actions 
                           don't migration audits with this action (Update, Creation, Delete)
  --include-audit-objectTypes 
                           migration only audits with this objectType (case, case_artifact, case_task, ...)
  --exclude-audit-objectTypes 
                           don't migration audits with this objectType (case, case_artifact, case_task, ...)

So, you can migrate your data directly to TheHive 4.0.1.

However, please note that the migration tool does not support data from TheHive 3.5.0 yet. In short, you can only consider migrating you data from TheHive 3.4.0, 3.4.1, 3.4.2 or 3.4.4 to TheHive 4.0 or TheHive 4.0.1 for the time being.

How to install/update ?

Our installation guides are up-to-date.

Important note

If you are updating an existing instance of TheHive 4.0.0, ensure to read the installation guide. Since our recent release of TheHive 3.5.0, we updated our repositories. The codename of repositories containing all new released packages is now named release, instead of stable. So update your apt or yum repository file.

Have you got a minute to let us know how you use TheHive ?

We would love to hear from you (10 questions < 1minute) : https://t.co/l5osjvQ1k7

Many thanks for taking the time 🙏

How to report issues

Please open an issue on GitHub with the dedicated template for TheHive 4. We will monitor them closely and respond accordingly. 

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Discord, or send us an email at support@thehive-project.org. We will be more than happy to help!

TheHive 3.5.0-RC1 and Cortex 3.1.0-RC1 are here!

Elasticsearch 7.0 can read indices created in version 6.0 or above. An Elasticsearch 7.0 node will not start in the presence of indices created in a version of Elasticsearch before 6.0.

https://www.elastic.co/guide/en/elasticsearch/reference/7.x/breaking-changes-7.0.html#_indices_created_before_7_0

Who could imagine what’s hiding behind this sentence ? 

To be honest, we managed to support Elasticsearch 7.x pretty quickly ! But only for new and recent installations and instances — read, initially installed with Elasticsearch 6.x. 

The harder part was ensure older instances, with indexes created  with Elasticsearch 5.x, can migrate smoothly like for previous migrations: «stop the application, update the database software, update application and restart everything ». You might have to put your hands on the keyboard.

Source: Google Images

⚠️  TheHive 3.5.0-RC1 and Cortex 3.1.0-RC1 are not recommended for production use. These versions are intended for test only ; please, read carefully the full blog post and the associated documentation. Feel free to try it, try your migration and send us your feedbacks. 

New and recent installations

If your instance has been initiated with Elasticsearch 6.x, you can follow the following process : 

  • Stop TheHive version 3.4.2
  • Stop Elasticsearch version 6.x
  • Update Elasticsearch configuration file
  • Update Elasticsearch to version 7.x and restart the service
  • Update TheHive and restart the service
  • Update Cortex and restart the service 

Instructions to install TheHive 3.5.0-RC1 or Cortex 3.1.0-RC1 can be found in this guide.

At this stage, connect TheHive and Cortex with your browser and you should be invited to update the database : 

Older indexes

This is the tricky part. If you are using an instance initiated with Elasticsearch older that version 6.0, it is highly probable that you have to follow an heavier process to upgrade. In few words, you will have to :  

  • Stop TheHive and Cortex applications
  • Create new indexes in Elasticsearch 6.x with part of the settings of your current indexes
  • Do specific reindexing operations to this new indexes
  • Delete old indexes.

How to identify if your index is ready for Elasticsearch 7

You can easily identify if indexes are ready for Elasticsearch 7. On the index named the_hive_15 run the following command:

curl -s http://127.0.0.1:9200/the_hive_15?human | jq '.the_hive_15.settings.index.version.created_string'

If the version is 6.x.x then the index will be read by Elasticsearch 7.8.x. Otherwise (version is 5.x.x of below), reindexing the index is required.

Migration guide

You are not left alone there. A dedicated documentation is available. It should help you run this specific actions on your Elasticsearch database, and also install or update application whether you are using DEB, RPM or binary packages, and even docker images : 

https://github.com/TheHive-Project/TheHiveDocs/blob/master/admin/upgrade_to_thehive_3_5_and_es_7_x.md

How to report issues

Please open an issue on GitHub using the template made for TheHive 3.x if you’d like to report a bug on this version. We will monitor those closely and respond accordingly. 

TheHive 4.0 is out!

Introduction

Several months,no, years ! after the first line of code – the first line was committed in 2016–, we are very excited and proud to announce the release of TheHive 4.0.

This means more than a major version for us. This was – and still is — like a completely new project, a new generation, a lot more challenging than before. We had to make the application climb a major step to introduce new key features, some we added in this version, others we have in mind for the future.

Objectives

The development of the second generation of TheHive, aka. version 4, was driven by three main objectives:

  • Add support to multi-tenancy: allow 1 instance of TheHive to serve many teams and organisations
  • Add support to Role Based Access Control to define fine grained user profiles
  • Rethink the data model and structure to support the goals listed above (Moving from Elasticsearch as main persistence layer, to a data model designed as a graph).

Challenges

TheHive Project is thoroughly adopted by SOC, CERTs and CSIRT teams, who decided to go with TheHive Project since the first releases. It is worth noting that until today, TheHive has had a total of 52 releases since 2016.

Those teams helped the project by contributing to our QA, questions, feature requests etc… and our way of thinking drove us to not let them down, and we decided to produce a backward compatible software.

The way we have been working until now aims to make our community move smoothly from TheHIve 3 to TheHive 4.

Backward compatibility

This is the most difficult challenge we have had, but we have hard heads and soft hearts.

TheHive 4 is expected to be backward compatible, thanks to APIs v0. Yes, we provide versioned APIs having the same endpoints as TheHive 3, and producing the same results. Search APIs also support the same query language, except some corner cases like searching using the “_string” operator (which is tightly coupled to Elasticsearch query language, but we have working alternatives).

Performance concerns

Supporting backward compatibility might force you to accept complex designs. And TheHive 4 RC3 was a clear example of that limitation.

Many kind users who tested TheHive 4 RC3, raised performance issues, slow UI problems etc… And it was completely expected. We thank them for making such a pressure on us, we used it to boost the refactoring of the UI, which was using backward compatible APIs (unoptimized for the new data model and representation), specially to read data (listing cases or observables for example).

We can discuss the technical details of this hard point later, but it mainly relates to navigating through graph-based data using a document based query system, which is not optimised.

For example, if you want to search for list of observable of a given case, the ideal way of doing that on a graph-base model is to:

  • Get the case by its ID, which is indexed (very fast operation)
  • Navigate through case relation, to find its links of type observable

But the backward compatible query language works differently: It scans all the graphs to search for observables that have a case parent with a given ID, which has a slower performance in a graph-based database.

Multi-tenancy and RBAC

TheHive 4 comes with a special multi-tenancy support. It allows the following strategies:

  • Use a siloed multi-tenancy: you can define many organisations, without allowing them to share data
  • Use a collaborative multi-tenancy: you can define a set of organizations and allow them to collaborate on specific cases/tasks/observables, using custom defined user profiles (RBAC)

This feature is very powerful but has a cost: an expected performance overhead. For example, when scanning the graph of data to search for a list of cases, TheHive must return the cases of your organisation and the case you can have access to because of the sharing rule.

New foundations

TheHive 3 was based on a framework called Elastic4play, written by Thomas to abstract all the routines required by a web application written with play 2 and using Elasticsearch.

TheHive 4 has its own core framework: Scalligraph, built to handle the following features.

Scalligraph will be the foundation of the next major version of Cortex.

What’s new in 4.0 

TheHive 4.0 release has a significant amount of changes. We will quickly explain the most important, and you can refer to the change logs if you need to have more details.

UI Performance

This was the most important task of this release. As we mentioned above, we were using backward compatible APIs in RC3 release, and migrated 80% of the UI to use the APIs v1 which are optimised for the new graph-based and multi-tenant data model.

OAuth2 Support

This topic gave birth to many github issues, some of them related to TheHive’s UI not correctly redirecting authenticated users. OAuth2 support has been tested with many providers like: Okta, Keycloak, FusionAuth, Microsoft Azure, Office 365 and Google Gsuite.

Starting from this version, there is an API endpoint that handle all the authentication and redirections: /api/ssoLogin

Here is a configuration sample for MS Office 365

{
  name: oauth2
  clientId: "CLIENT_ID"
  clientSecret: "CLIENT_SECRET"
  redirectUri: "http://THEHIVE_URL/api/ssoLogin"
  responseType: code
  grantType: "authorization_code"
  authorizationUrl: "https://login.microsoftonline.com/TENANT/oauth2/v2.0/authorize"
  authorizationHeader: "Bearer "
  tokenUrl: "https://login.microsoftonline.com/TENANT/oauth2/v2.0/token"
  userUrl: "https://graph.microsoft.com/v1.0/me"
  scope: ["User.Read"]
  userIdField: "mail" 
}

You can find more details about the OAuth2 support in the authentication config documentation

Improved Analyzer and Responder selection

Analyzer selection when calling bulk observable analysis has been improved to show the possible analyzers per observable type.

Analyzers selection during observable bulk analysis

For responders, the user experience has been improved as well, especially for instances with a big number of responders. The simple dropdown menu available to select responders has been replaced by a dialog allowing list filtering and scrolling:

New Responder selection dialog

Add bulk operations to case listing

Before this release, simple case updates required visiting the cases one by one and editing them. We added in this release a bulk edit feature, depending on user’s permissions on the selected cases

Bulk edit dialog, used here from case list

The same bulk editing component has been used to improve the same operations on observable list page.

Other noteworthy changes

We need to mention that the following changes have been included in TheHive 4.0 release:

  • Add pagination and filtering to users administration
  • Add back the UI configuration by organisation. The only available option is related to enabling/disabling the use of Empty Case.
  • Show sharing summary in task and observable lists
  • Improve alert preview dialog
  • Add alert externalLink feature allowing the display of external links for any alert, not only MISP alerts.

Known limitations

Even after 49 closed Github Issues, there are still major topics to be addressed by the upcoming releases:

  • Add back support to case merge which is not satisfying today. The challenge is to find the best to merge cases and make sure that it works in a profile-based multi-tenant design.
  • Add full text search support. In older versions, TheHive benefited from the full text search capabilities of Elasticsearch. With the new database and persistence system, full text support requires adding a dedicated indexing layer.

Installing and testing TheHive 4.0

After months of testing versions, this official release means that we consider it ready for production purposes. If you’re new with TheHive, we recommend going with TheHive 4.0.

Several installation guides have already been published, suitable with the chosen operating system and installation type, and new are coming.

For testing and training purposes, a virtual machine with a simple configuration of TheHive 4.0 and Cortex 3.0.1, is also published and available starting from now. Please refer to the documentation for download and usage instructions.

Want to upgrade from TheHive 3.x ?

All changes brought to TheHive make the upgrade more challenging than installing the new package and watch the progress bar. To support you with the upgrade, a migration tool comes along with the application to shift your current version of TheHive to TheHive 4.0.

A dedicated guide has been published to help users with this significant task. We recommend using a new server aside from your production server to ensure everything works fine with the migration.

Future of TheHive 3.x

This major outcome doesn’t mean TheHive 3 end of life is reached. As previously announced, we plan to support this version for some time, our next milestone being to support Elasticsearch 7.x with a first Release Candidate.

How to report issues

Please open an issue on GitHub using the template made for TheHive4 if you’d like to report a bug on this version. We will monitor those closely and respond accordingly.

Cortex-Analyzers 2.4.0: 138 Ways to Analyze, 10 Methods to Respond

Guess what? TheHive Project is still alive and well, as Saâd already mentioned in a previous blog post.

We’ve been certainly very busy lately, preparing the upcoming release of TheHive 4 and doing many other things beside working on our FOSS project. As a result, it took us a rather long time to merge several community contributions and reduce the sizeable pile of pull requests.

We would like to thank our contributors for their patience and we hope the cyberdefenders out there will enjoy the brand new Cortex-Analyzers 2.4.0 release, with many new analyzers, responders and some bug fixes & improvements, bringing the total to a whooping 138 analyzers (counting all flavors) and 10 responders!

Additionally, with this release, all analyzers are now using Python 3. No more Python 2 technodebt!

Photo by Saâd Kadhi

What’s New?

New Analyzers

8 new analyzers have been added to this release:

1 analyzer has new flavors:

New Responders

3 new responders have been added:

Overview of the New Analyzers

DomainToolsIris

This analyzer looks up domain names, IP addresses, e-mail addresses, and SSL hashes using the popular DomainTools Iris service API.

The analyzer comes in 2 flavors:

  • DomainToolsIris_Investigate: use DomainTools Iris API to investigate a domain.
  • DomainToolsIris_Pivot: use DomainTools Iris API to pivot on ssl_hash, ip, or email.

A valid DomainTools API integration subscription is needed to run this analyzer.

TheHive displays the analyzer results as follows:

DomainToolsIris short report
DomainToolsIris long report

EmailRep

The EmailRep analyzer checks the reputation of an email address against the emailrep.io database.

IPInfo

This analyzer accesses IP-centric features provided by ipinfo.io. While the EmailRep API can be used without a token for limited usage, the ipinfo.io analyzer requires the configuration of an API token before use.

Maltiverse

This analyzer lets you query the free Maltiverse Threat Intelligence platform for enrichment information about a particular hash, domain, ip or url.

TheHive displays the analyzer results as follows:

Maltiverse short report
Maltiverse long report

MalwareClustering

Andrea Garavaglia contributed this one a long time ago and we finally merged it into the Cortex-Analyzers repository. Andrea gave a talk about the background of this analyzer at the fourth MISP summit. You can watch it here.

In order to use the analyzer, you need to point it to a Neo4j server (you need to supply the host, port, login & password).

PaloAlto Autofocus

This analyzer lets you leverage PaloAlto Autofocus services. Provided you are an Autofocus customer and you have access to their API, you need to configure the analyzer with your username and a token key.

The analyzer comes with 3 flavors:

  • AUTOFOCUS_GetSampleAnalysis lets you request a full report for a given hash.
  • AUTOFOCUS_SearchIOC lets you research for samples linked to specific IoCs with datatypes like domain, fqdn, user-agent, imphash, ip, mutex, tag and url. Please note that mutex and tag are not default datatypes in TheHive. You need to create them in TheHive before you can leverage them.
  • AUTOFOCUS_SearchJSON lets you research for samples based on a complex JSON query.

Important: TheHive has no templates corresponding to this analyzer have been published yet. They will be provided in the near future.

SpamhausDBL

This analyzer performs reputation lookups of a domain or a fqdn against Spamhaus Domain Block List (DBL).

TheHive displays the analyzer results as follows:

SpamhausDBL short report
SpamhausDBL long report

TeamCymruMHR

This analyzer queries Team Cymru’s Malware Hash Registry for known malware hashes (MD5 or SHA-1). If it is malware and known by the service, it returns the last time it has been seen along with an approximate anti-virus detection percentage.

Overview of the New Responders

KnowBe4

This responder allows the integration between TheHive/Cortex and KnowBe4’s User Events API.
If a mail observable is tagged with a specified tag, corresponding to the responder’s configuration (e.g. phished), then the associated user will have a custom event added to their profile in KnowBe4.

A valid account on KnowBe4 and an API key are required to run this responder.

Minemeld

This responder sends observables you select to a Palo Alto Minemeld instance.

To run this responder, a MineMeld Threat Intelligence Sharing account is needed.

Wazuh

This responder performs actions on Wazuh, the open source security monitoring platform. It currently supports ad-hoc firewall blocking of ip observables.

Improvements

New PassiveTotal flavors

Thanks to Brandon Dixon, the PassiveTotal analyzer gains 3 new flavors, bringing the total to 11:

  • PassiveTotal_Trackers let you make tracker lookups on observables of type domain, fqdn and ip.
  • PassiveTotal_Host_Pairs let you make host pair lookups on observables of type domain, fqdn and ip.
  • PassiveTotal_Components lets you make components lookup on observables of type domain, fqdn and ip.

They come with their own report templates.

GreyNoise Analyzer

The analyzer has been updated to support GreyNoise API v2, thanks to the contribution of Whitney Champion (#562).

New Data Types Supported by Some Analyzers

  • VirusTotal_GetReporthas been updated to allow requests for observables of type fqdn.
  • Threatcrowd has been updated to allow requests for observables of type domain.
  • Shodan has been updated to allow requests for observables of type fqdn.

Fixes

  • [#602] The MISP analyzer was bumped to version 2.1 and is ready to use PyMISP 2.4.120.

Get It While Supply Lasts!

I’m Hype

If you are using the dockerized analyzers & responders, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button.

I’m Country

If you are still using the old-style way of installing analyzers and responders, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

TheHive 3.4.0 & Cortex 3.0.0 Released

For many months, we have been concentrating our efforts on TheHive 4, the next major version of your favourite Security Incident Response Platform, which we’ll finally provide RBAC (or multi-tenancy if you prefer), a feature that Cortex had for quite some time now.

Source : dilbert.com © Scott Adams

As you well know, both TheHive and Cortex rely on Elasticsearch (ES) for storage. The choice of ES made sense in the beginning of the project but as we added additional features and had new ideas to give you the best experience possible, we faced several ES quirks and shortcomings that proved challenging if not outright blocking for making our roadmap a reality, including RBAC implementation in TheHive, a far more complex endeavour than RBAC in Cortex. Transitioning from ES to graph databases was necessary and since we want our existing users to have a smooth migration path, TheHive 4 (the first release candidate should come out of the oven by the end of the year) will support both ES and graph databases.

But while we were focusing on that, we completely lost sight of the end of life of ES 5.6 so we wrote an apology to you, our dear users, back in May.

Shortly after, we released TheHive 3.4.0-RC1, to add support for ES 6 (with all the breaking changes it has introduced). We also did the same for Cortex with the release of Cortex 3.0.0-RC3. We also took that opportunity to clear out some AngularJS technodebt we had.

We then asked you to take them for a spin and report back any bugs you find given that both versions had to support ES 5.6 and ES 6 to allow for proper migration.

After a few rounds of release candidates, we are pleased to announce the immediate availability of TheHive 3.4.0 and Cortex 3.0.0 as stable releases.

Before upgrading your existing software to these new versions, please make sure to read the blog post we wrote back in June. We invite you to pay great attention to the regressions that we were forced to introduce because of ES 6.

You should also note that, in addition to ES 6 support, Cortex 3.0.0 supports fully dockerised analyzers and responders. We’ll elaborate on this in a future blog post soon.

Changelogs

If you are interested in some nitty-gritty details, we invite you to read the relevant changelogs since our last post on the subject:

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help as usual!

DigitalShadows Feeder 2.4.5

DigitalShadows2TH 2.4.5, a new version of the DigitalShadows feeder for TheHive, has been released with major improvements.

Source : https://www.bustle.com

Dockerise All The Things!

The most notable one is that you can now use it with docker. Run docker pull thehiveproject/ds2th and create a homedir for configuration and logs.

$ DS2TH_HOMEDIR = /opt/thehive_feeders/Digitalshadows2TH/
$ docker pull thehiveproject/ds2th:latest
$ mkdir -p $DS2TH_HOMEDIR/{config,log}
$ wget -O $DS2TH_HOMEDIR/config/config.py\
https://raw.githubusercontent.com/TheHive-Project/DigitalShadows2TH/master/config.py.template

Edit the config.py before runing the docker command below:

$ docker run --rm --net=host --mount\
type=bind,source="$DS2TH_HOMEDIR"/config,target=/app/config --mount type=bind,source="$DS2TH_HOMEDIR"/log,target=/app/log certbdf/ds2th <OPTIONS>
Quick Installation

If that sounds still complicated, worry not! We also wrote a little program that helps you prepare the environment and install everything. We ensured that it works well on Ubuntu 18.04. The program uses two environment variables to set up everything: FEEDERS_SYSACCOUNT and FEEDERS_HOMEDIR :

$ export FEEDERS_SYSACCOUNT=thehive
$ export FEEDERS_HOMEDIR=/opt/thehive_feeders

There are also sane, default settings in case you did not set any value. DigitalShadows2TH’s home directory will be set to /opt/thehive_feeders/DigitalShadows2TH. To use the script, run the following command line and follow the instructions:

 $ wget -qO- https://raw.githubusercontent.com/TheHive-Project/DigitalShadows2TH/mater/INSTALL/install_with_docker.sh | sudo -E bash -

Custom Case Templates

Previous versions of DigitalShadows2TH allowed only one case template to be associated with alerts created by the feeder in TheHive. Starting from DigitalShadows2TH 2.4.0, you can define a case template for each type of incidents raised by DigitalShadows in the configuration file.

The configuration pertaining to TheHive looks as follows:

TheHive = {
 'proxies':{
     'http': '',
     'https': ''
},
   'url':'THEHIVE_URL',
   'key':'THEHIVE_API_KEY',
   'templates': {
       'default':''
  }
}

Each incident type in DigitalShadows can be associated with a case template in TheHive, for example:

TheHive = {
  'proxies':{
      'http': '',
      'https': ''
  },
    'url':'THEHIVE_URL',
    'key':'THEHIVE_API_KEY',
    'templates': {
        'default':'MY_DEFAULT_CASE_TEMPLATE_FOR_DIGITALSHADOWS_INCIDENTS',
        'DATA_LEAKAGE': 'MY_DATA_LEAKAGE_CASE_TEMPLATE',
        'CYBER_THREAT': 'MY_CYBER_THREAT_CASE_TEMPLATE'
    }
}

A template can be defined for all the following DigitalShadows incident types:

  • DATA_LEAKAGE
  • CYBER_THREAT
  • PHYSICAL_SECURITY
  • SOCIAL_MEDIA_COMPLIANCE
  • BRAND_PROTECTION
  • INFRASTRUCTURE

A default template can be defined for DigitalShadows incidents. If no template is found for a specific incident type, the feeder looks for the default template. if no default template is found, an empty case will be created by when importing the alert.

Update or Install

If you are not using docker, just pull the repository and update your configuration file with the new templates part for TheHive.

Update your Repository
$ cd /opt/TheHive_feeders/DigitalShadows2TH/
$ git pull

The configuration file has changed, so you need to update yours before running the program. A new templates section has been added for TheHive and the path has changed. It is now in the config/ directory of the project.

Install and Use via the Code Repository
$ cd /opt/TheHive_feeders 
$ git clone https://github.com/TheHive-Project/DigitalShadows2TH.git

After that, follow the prerequisites and edit the configuration file. In /opt/TheHive_feeders/DigitalShadows2TH/config/ copy config.py.template to config.py and modify it.

Use cases and detailed configuration instructions can be found in the README file in the repository.

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

More than 120 ways of Analyzing your Observables

There’s a new version of Cortex-Analyzers in cybertown and its has an awesome, mind-blowing name and that’s… wait for it… wait for it: 2.1.0.

In this new release, we added two analyzers which bring the total number to more than 120:

We could not duly test DNSSinkhole since we do not have access to the associated service. So we would really appreciate it if you could test it and let us know whether it works or not.

Others analyzers have been fixed or improved:

DNSSinkhole

This analyzer lets you check if an IP address has been registered in your DNS sinkhole. TheHive displays the analyzer results as follows:

DNSSinkhole analyzer: long report
DNSSinkhole analyzer: short report

TalosReputation

This analyzer lets you determine whether an IP address has been reported as a threat on Cisco Talos Intelligence service. No special access to the service is required to run the analyzer.

TheHive displays the analyzer results as follows:

Talos Reputation: long report
Talos Reputation analyzer : short report

Crt.sh

This analyzer has been enriched to display SHA-1 fingerprints. The long report format has been updated to reflect this new information.

Crt.sh analyzer: long report

FileInfo

FileInfo has been updated and is now able to parse PDF files and extract IOCs such as URLs, hosts, domains, IPs, hashes and many more.The analyzer does also support the last version of the extract-msg library.

FileInfo analyzer: IOC Parser long report
FileInfo analyzer: IOC Parser short report

VirusTotal and Python3

The VirusTotal analyzer, including all its flavours, now uses Python3 and an updated virustotal-api library.

Yeti API key

An optional API key can now be configured and used by the Yeti analyzer.

Malwares_GetReport

A hash computation has been fixed in this analyzer.

EMLParser

A first fix has been introduced to avoid this analyzer to crash when there is no content-description in content_header, and a second has been added to correct a header display issue.

IBM XForce Lookup

The analyzer has been improved to allow users to add a trailing / at the end of the API URL without breaking everything.

Updating your Analyzers in Cortex 2.x

Each analyzer and responder comes with its own, pip compatible requirements.txt file. Run the following commands to update your Cortex analyzers to the latest version:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip2 install -U -r $I || true; done && \
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, do not forget to login to Cortex as an orgadmin and click on the Refresh Analyzers button. Refer to the online Cortex documentation for further details.

Dockerised Analyzers

Cortex 3.x gives you the opportunity to run dockerised analyzers and responders. This means that you no longer have to download all the git repository of Cortex-Analyzers and run lengthy commands to update your analyzers and responders.

If you want to use dockerised analyzers and responders, ensure that the URL of the catalog.json file corresponding to the Cortex-Analyzers repository is registered in application.conf. Please note that this won’t work if you are tracking the stable catalog.

After doing so, do not forget to login to Cortex as an orgadmin, click on the Refresh Analyzers button, then Disable and Enable again each analyzer and responder. Analyzer (and responder) updates should occur automatically as long as docker.autoUpdate is set to true in application.conf (this is the default setting).

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!