TheHive Project’s chat has a new home

TheHive Project an Discord

Back in April 2020, we discovered an issue in our GitHub repository opening the discussion about moving our chat and creating a discord server for community discussions. It was obviously a relevant suggestion, but one we couldn’t address at the time since we were head down focusing on developing product updates (TheHive 4, TheHive and Cortex with ES7 support) and building a sustainable structure for your beloved TheHive Project to prosper in the long run.

Today, our public community has strong foundations and keeps growing as we welcome new users and organisations wishing to start their journey with TheHive Project. We can count on an amazing number of trusted and invested users willing to share their knowledge and help newcomers (the current Gitter channel is mostly self-managed). The GitHub issue highlighted several key improvements we could work on to provide a better chat experience to our community:

  • offering different channels, for different products to handle more targeted questions
  • implementing new roles like moderators and trusted-users
  • creating a channel for announcements
  • creating channels for contributions, automation
  • creating channels for languages other than English
  • listening to our member’s requests to improve the community server

We truly wish to enhance the chat experience of all community members, whether they join the conversation to ask a question, share their experience or help others troubleshooting an issue.

Why Discord?

Let us share a story.

TheHive Project’s core team first used Slack as its main private chat and communication tool. Slack is fine but we had some frustrating experience with it (ex: limited search history). We then moved to Keybase which is still our daily internal chat platform. Although we are satisfied with Keybase, we felt the onboarding experience might be a little too intense for some users.

So we decided to have a look at Discord, which was initially designed for gamer, allowing text chat, video, voice calls and screen sharing. Discord has great mobile apps too, in addition to those for web and desktop. And it’s really fast! It’s also getting a strong footprint with open source projects.

Last week, we posted a poll on Twitter about the move, and here are the democratic answers:

Twitter poll about Discord vs. Gitter

This confirmed our impression, so let’s do it.

How to join?

It’s easy peasy, here is the link Join our new Discord based community. It requires a Discord account with a valid email address (which is the lowest requirement). You will be welcomed with a screen that reminds the rules and code of conduct.

Welcome screen

Once registered:

  • If you wish to introduce yourself, you can share your story with us through the `#introductions` channel.
  • If you wish to share something you built in top of our products, your can do it in the `share-your-work`
  • If you need a new channel dedicated to your language, you can just ask for it or reach any core member or moderator
Community rules

What’s next?

We hope the current 1k+ users registered in Gitter will migrate to our new Discord platform. We will keep the light on in the Gitter channel for the time being, but we hope the Discord community will be the new land for all of you.

It’s not Patch Friday… TheHive 4.0.2 released

Last Friday, our team released a significant number of changes and fixes, improving TheHive 4. The community was very reactive and hurried to test it. Today we are announcing a patch release to fix annoying issues we decided to quickly patch: welcome to 4.0.2.

The major issue is related to Alert bulk merging which is part of the big refactoring of the Alert listing section.

Well, s**t happens, but as a gentlemen, we owe you a feature, or two 🙂

What’s new?

In addition to the Alert bulk merging issue, we fixed bugs related to:

  • Migration
  • Filtering observables by attachement attributes
  • Backward compatible APIs

Curious to read the detailed change log? Here your go.

TheHiveFS, do you remember?

It stands for TheHive File System, a feature we released with TheHive 4.0-RC1. In today’s milestone we are improving the security of this feature by adding a new user permission, required to have access to TheHiveFS features.

The newly created accessTheHiveFS permission is included by default in both org-admin and analyst user profiles.

Refined editable fields, for a better UX

In TheHive UI, editing objects relies on editable fields instead of dedicated form to edit objects. This means you can update a case title by clicking the Blue Pencil icon displayed when you mouse-over the case title for exemple.

In this release we refactored all the editable fields to provide a better user experience:

  • You no longer need to click just on the `Blue Pencil` icon to switch to edit mode, you just need to click on any value, for example the Case assignee field, or the Observable tags field:
Editable fields on mouse over events

Editable fields have now a `Clear` option, allowing users to unset the value of an attribute:

Editable custom fields can be set empty

This new improvement benefits to the custom fields sections in Case and Alert details sections, as showcased above.

Configurable layout of custom fields

In older TheHive versions, custom fields were displayed using a single column. Cases with big number of custom fields produced a long scrolling Case details pages, so we decided in TheHive 4 to use a 3-column layout to reduce the resulting scroll fatigue.

Now some users are complaining because of long custom fields values not being correctly displayed. So we decided to let users choose their preferred layout.

1-column layout to display custom fields in a Case details page
2-columns layout

How to install/update ?

Our installation guides are up-to-date.

Important note

If you are updating an existing instance of TheHive 4.0.0, ensure to read the installation guide. Since our recent release of TheHive 3.5.0, we updated our repositories. The codename of repositories containing all new released packages is now named release, instead of stable. So update your apt or yum repository file.

Have you got a minute to let us know how you use TheHive ?

We would love to hear from you (10 questions < 1minute) : https://t.co/l5osjvQ1k7

Many thanks for taking the time 🙏

How to report issues

Please open an issue on GitHub with the dedicated template for TheHive 4. We will monitor them closely and respond accordingly. 

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Discord, or send us an email at support@thehive-project.org. We will be more than happy to help!

It’s now time for TheHive 4 to get an update: TheHive 4.0.1 is out!

Back in July, we announced TheHive 4.0, the foundation for future releases of your favorite Open Source SIRP.

Today, we are pleased to deliver one of the biggest patch releases we’ve done so far, in terms of number of fixed issues: TheHive 4.0.1! Not only did we squash lots of bugs, we also included some cool improvements we didn’t want to hold until the next version.

As for TheHive 3 and Cortex unveiled few days ago, we fixed the vulnerabilities reported by Play framework in October as well.

A huge work has been accomplished to fix a backlog of bugs (thanks to the community) in several parts of the product:

  • Configuration;
  • API;
  • Dashboards;
  • Livestream;
  • Cases;
  • Alerts;
  • Observables;
  • Multi Factor authentication;
  • Active Directory authentication;
  • Export to MISP.

You can find details of the 80+ issues included in this milestone: Changelog file.

Wait, there is also new stuff:

  • Cases and Alerts similarity fine tuning;
  • Similar cases in Alert preview panel refined;
  • Filtering capabilities improvements;
  • New UI settings & options;
  • Custom fields in Case and Alert lists;
  • Statistics panel improvements;
  • API key in user settings page;
  • More migration tool options.

Cases and Alerts similarity fine tuning

In TheHive, correlation between Alerts and Cases, and between Cases themselves, is computed throughout their respective Observables:

  • A Case is considered similar to an Alert, if they have at least one common observable;
  • A Case is considered related to another Case, if they have at least one common observable.

In some situations, those relationships are undesirable when based on useless/meaningless observables.

Starting with TheHive 4.0.1, Observables are enriched with a new attribute called ignoreSimilarity, set to false by default. Now, when this attribute is set to true, an observable is simply not taken into account in the relationship between two cases, and is not involved in the similarity calculation, for Cases and Alerts.

This attribute is displayed alongside with the IOC and Sighted flags, in the Observables list – which has been tweaked for the occasion – as well as in the detailed view.

IgnoreSimilarity flag in the revamped Observables list

And you can also edit this for a set of selected observables.

Edit “Ignored for similarity”

Long story short, if you want to break a correlation, you just need to set the linking Observable‘s ignoreSimilarity flag to true.

Alerts and similar Cases

The attribute mentioned earlier is taken into account when correlating Alerts with existing Cases : the Similar Cases view in the Alert preview dialog.

Observables with ignoreSimilarity set to true will be ignored when fetching cases similar to the current Alert, whether it is in the Alert or the Case side.

A forthcoming update of thehive4py library will allow to set this attribute in Alerts, along with many other new things.

The Similar Cases view has been thoroughly enhanced to provide more highlights facilitating alert triage and the merging of alerts into existing cases. First, a column named “Matched” has been added. It contains the list of observable dataTypes and the number of occurrences matching the Alert : this should be very useful to identify quickly what type of data has matched the alert content. A filter on this column has also been added.

And there are more refining capabilities in this view:

  • Sorting and Filtering on Title of Similar Cases;
  • Specifying a minimum observables rate that matches the Alert;
  • Specifying a minimum IOCs rate that matches the Alert.
Refining the Similar Cases view during Alerts triage

… And, above all, columns can now be sorted. With all these new capabilities, triage and merge Alerts in Cases should be easier.

Filtering capabilities improvement

In TheHive 4, filter forms have the same structure all over the product. Any improvement in the area, benefits to all the listing pages.

For example, in the alert similar cases tab, a new filter form has been included. You can quickly select the type of Cases you want to display, but also use the advanced rules and search criterias.

In this release, the filter component has new capabilities:

  • Date filter: now allows setting dynamic ranges instead of setting from and to fixed date values. Possible values are:
    • Today;
    • Last 7 days;
    • Last 30 days;
    • Last 3 months;
    • Last 6 months;
    • Last year;
    • Custom: allowing the old-fashioned way of setting date ranges.
  • Tag filter: Autocomplete is now enabled.
Advanced filtering for Similar Cases during Alert triage
Autocompleted tags when adding a filter

In TheHive 4.0.1, there is one more place that welcomed a filter form: the list of Alert observables in the preview dialog.

New Org level UI settings

UI configuration page is the place where TheHive adds new options to customise the behavior of the UI, for a given organisation. This is an area that will be expanded over time.

For this milestone, two new options have been introduced:

  • Allow merging Alerts in resolved cases;
  • Default filter of Alert similar cases.
Default filter for Similar Cases in Alerts preview

The first one is self-explanatory and will hide the `Merge into case` button in the Alert similar cases section

The second one allows an administrator define, for an organisation, the default filter to apply to Alert similar cases view. For example: “show only potential similar cases created in the last 7 days and are still open”.

If during the triage, the filter has been updated by an analyst, the default filter cas be retrieved through Default filter option of the Quick Filters.

Retrieve default filter in alert similar cases

Custom fields in Case and Alert lists

“Damn, why I’m not able to see my custom fields in my cases list”, “It’s a shame I cannot filter my alerts with my custom defined properties”… We are pretty sure you said it, at least once.

Well, you dreamt of it, and here we go, we did it for Cases and Alerts

  • You will be able to see all the custom fields you defined in your Case Templates and propagated to your Cases;
  • Custom fields with undefined values are not displayed;
  • Custom fields are shown with respect of the order you define;
  • Clicking on a custom field, filters your lists using the selected value;
  • You can enable displaying the custom fields using the toggle button on the toolbar located on top of the filter form;
  • All those capabilities are available in Cases and Alerts views.
Custom Fields in Alerts list

New statistics view in Cases and Observables lists

The Stats view has been revamped to represent the statistics using both tables and shiny colourful donuts (not only for managers).

This improvement has been made in Cases, Observables and Alerts views.

New “Stats view” in Cases list

User settings

This feature has been initially introduced in Cortex and was missing in TheHive. All users can now access their API key, copy or update it, as long as an administrator created it first. No more API keys sent over emails or chat applications.

User settings with API key enabled

Migration

The migration assistant program has been updated with a few bug fixes and new filtering options. When migrating, you can now specify:

  • If you want to include or exclude Alerts generated from specific Sources or by their Type;
  • If you want to include Audit trails regarding specific objectType (case, case_task, case_observable …);
  • If you want to filter out Audit trails with specific actions (Update, Creation, Delete).
/opt/thehive/bin/migrate --help

[..]  
  --include-alert-types ,...
                           migrate only alerts with this types
  --exclude-alert-types ,...
                           don't migrate alerts with this types
  --include-alert-sources ,...
                           migrate only alerts with this sources
  --exclude-alert-sources ,...
                           don't migrate alerts with this sources
[..]   
  --include-audit-actions 
                           migration only audits with this action (Update, Creation, Delete)
  --exclude-audit-actions 
                           don't migration audits with this action (Update, Creation, Delete)
  --include-audit-objectTypes 
                           migration only audits with this objectType (case, case_artifact, case_task, ...)
  --exclude-audit-objectTypes 
                           don't migration audits with this objectType (case, case_artifact, case_task, ...)

So, you can migrate your data directly to TheHive 4.0.1.

However, please note that the migration tool does not support data from TheHive 3.5.0 yet. In short, you can only consider migrating you data from TheHive 3.4.0, 3.4.1, 3.4.2 or 3.4.4 to TheHive 4.0 or TheHive 4.0.1 for the time being.

How to install/update ?

Our installation guides are up-to-date.

Important note

If you are updating an existing instance of TheHive 4.0.0, ensure to read the installation guide. Since our recent release of TheHive 3.5.0, we updated our repositories. The codename of repositories containing all new released packages is now named release, instead of stable. So update your apt or yum repository file.

Have you got a minute to let us know how you use TheHive ?

We would love to hear from you (10 questions < 1minute) : https://t.co/l5osjvQ1k7

Many thanks for taking the time 🙏

How to report issues

Please open an issue on GitHub with the dedicated template for TheHive 4. We will monitor them closely and respond accordingly. 

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Discord, or send us an email at support@thehive-project.org. We will be more than happy to help!

New releases for TheHive and Cortex: Elasticsearch 7 support and security fixes.

TheHive and Cortex with ES7 support

We are happy to announce the immediate availability of TheHive 3.5.0 and Cortex 3.1.0 that supports Elasticsearch 7. We are also releasing TheHive 3.4.4 to include security upgrades. All of them are including fixes for vulnerabilities reported on Play Framework this month. We encourage you to upgrade.

As promised, despite the release of TheHive 4.0 in July, we are still support version 3. Today we are releasing two versions of TheHive 3, but why ?

As mentioned previously, the EOL of Elasticsearch version 6 is the reason why we decided to upgrade TheHive and Cortex to exclusively support Elasticsearch version 7.

If you want to be up-to-date with TheHive and Cortex, you must use Elasticsearch 7 and the new released versions of our products: TheHive 3.5.0 and Cortex 3.1.0.

With that being said, we won’t let down the users who cannot migrate their Elasticsearch immediately to version 7, so we decided to fix an embarrassing bug related to alerts with large amount of observables, Thanks to TheHive 3.4.4.

What’s new in TheHive 3.5.0 and Cortex 3.1.0

In addition to support for Elasticsearch 7.x, following fixes has been added in TheHive 3.5.0:

  • Fix a bug with the admin page of Analyzers report templates (#1591)
  • Responder list is ordered alphabetically (#1564)
  • Keep date filters when pivoting from Dashboards to search page (#1581)
  • UI Configuration option to choose to filter TAG1 AND TAG2 or TAG1 OR TAG2 in Alerts view (#1171)
  • Fix issue when clicking on Analyzers short reports (#1350)

In addition to support for Elasticsearch 7.x, following fixes has been added in Cortex 3.1.0:

  • Take into account defaultValue in Neurons flavor file (#309)

Oauth2

Use OAuth2 with TheHive 3.5.0 and Cortex 3.1.0

Both versions have been updated to improve OAuth2 authentication support. They are now working the same way than TheHive 4.0.0, with a quite similar configuration.

We invite you to refer to the documentation for each application to configure it: TheHive and Cortex

Our support on TheHive and Cortex

However, starting from now, we will no longer support TheHive and Cortex version that use Elasticsearch < 7: i.e. TheHive < 3.5.0 and Cortex < 3.1.0. So please make sure to update your instances and rely on up-to-date and supported components.

Be aware that:

  • Any issue reported in TheHive version 3.4.4 and lower, will be fixed on top of TheHive 3.5;
  • Any issue reported in Cortex version 3.0.1 and lower, will be fixed on top of Cortex 3.1.0.

This situation made us also add strong changes regarding our repositories for DEB and RPM packages. Read carefully what follows and find your situation to learn how to upgrade.

You are still using or plan to continue with Elasticsearch 6 ?

Upgrade to TheHive 3.4.4

  • apt update && apt install thehive if you are using debian subsystems;
  • yum install thehive if you are using RedHat, Fedora or CentOS.

If you are using docker image you need to specify the version. Get it by running the following command line:

  • docker pull thehiveproject/thehive:3.4.4-1

This version introduces a bug fix regarding the import of alerts having significant amount of observables.

Keep Cortex 3.0.1

3.0.1 is the last version of Cortex supporting Elasticsearch 6.x. So keep this version until you move to Elasticsearch 7.x.

You are using or plan to move to Elasticsearch 7.x ?

⚠️ DO NOT run an upgrade command on your system until your data has been migrated in Elasticsearch 7.x and Elasticsearch is running.

Upgrading an existing installation ?

Elasticsearch 7.x introduced changes that break our way of representing the data, so some updates need to be applied on the database configuration and on the index first.

We highly recommend reading carefully our dedicated migration guides before starting the upgrade process:

Obviously, we recommend testing this process on a testing environment before running it in production.

Running a fresh installation ?

To publish packages supporting Elasticsearch 7 and avoid anyone break his servers, we decided to create dedicated packages repository. To install TheHive 3.5.0, according to your Operating System, run the following processes.

Deb packages

After installing Elasticsearch 7.x, ensure your /etc/apt/source.list.d/thehive-project.list looks like this:

deb https://deb.thehive-project.org release main

Then, run following commands to install TheHive 3.5.0:

apt update
apt install thehive # or apt install thehive=3.5.0-1

and following commands to install Cortex 3.1.0:

apt update
apt install cortex # or apt install cortex=3.1.0-1
RPM packages

After installing and running Elasticsearch 7.x, ensure your /etc/yum.repo.d/thehive-project.repo looks like this:

[thehive-project]
enabled=1
priority=1
name=TheHive-Project RPM repository
baseurl=https://rpm.thehive-project.org/release/main
gpgcheck=1

Them, run following commands to install TheHive 3.5.0:

yum update
yum install thehive # or yum install thehive-3.5.0-1

And following commands to install Cortex 3.1.0:

yum update
yum install cortex # or yum install cortex-3.1.0-1
Docker image
  • TheHive 3.5.0:
    • docker pull thehiveproject/thehive:3.5.0-1
  • Cortex 3.1.0:
    • docker pull thehiveproject/cortex:3.1.0-1
Ressources

Again, we strongly invite you to read detailed installation instructions:

Docker image of Cortex 3.1.0

Important modifications have been introduced in the docker image of Cortex 3.1.0. This image does not come anymore with programs of Analyzers and Responders and their dependencies.

Cortex is able to run those programs with Docker when images exist. The default configuration included in the official docker image of Cortex uses our catalogs of images of Analyzers and Responders.

Running Analyzers and Responders directly in Cortex container (using “process” method) is still supported. You can include them in container thanks to the Docker volumes when you start the container. If they need dependencies, you can create your own Docker image from our official Cortex image. Below an example of Dockerfile that retrieves Analyzers and Responders like previous Cortex Docker image:

FROM thehiveproject/cortex:3.1.0-1
RUN apt-get update
RUN apt-get install -y --no-install-recommends                        \
        python-pip python2.7-dev python3-pip python3-dev              \
        ssdeep libfuzzy-dev libfuzzy2 libimage-exiftool-perl          \
        libmagic1 build-essential git libssl-dev dnsutils iptables
RUN pip2 install -U pip setuptools
RUN pip3 install -U pip setuptools
RUN git clone https://github.com/TheHive-Project/Cortex-Analyzers.git \
        /opt/Cortex-Analyzers
RUN for I in $(find /opt/Cortex-Analyzers -name 'requirements.txt')   \
    do                                                                \
        pip2 install -r $I || true                                    \
        pip3 install -r $I || true                                    \
    done

How to report issues

Please open an issue on GitHub if you’d like to report a bug for TheHive or Cortex. We will monitor those closely and respond accordingly. 

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Discord, or send us an email at support@thehive-project.org. We will be more than happy to help!

TheHive 3.5.0-RC1 and Cortex 3.1.0-RC1 are here!

Elasticsearch 7.0 can read indices created in version 6.0 or above. An Elasticsearch 7.0 node will not start in the presence of indices created in a version of Elasticsearch before 6.0.

https://www.elastic.co/guide/en/elasticsearch/reference/7.x/breaking-changes-7.0.html#_indices_created_before_7_0

Who could imagine what’s hiding behind this sentence ? 

To be honest, we managed to support Elasticsearch 7.x pretty quickly ! But only for new and recent installations and instances — read, initially installed with Elasticsearch 6.x. 

The harder part was ensure older instances, with indexes created  with Elasticsearch 5.x, can migrate smoothly like for previous migrations: «stop the application, update the database software, update application and restart everything ». You might have to put your hands on the keyboard.

Source: Google Images

⚠️  TheHive 3.5.0-RC1 and Cortex 3.1.0-RC1 are not recommended for production use. These versions are intended for test only ; please, read carefully the full blog post and the associated documentation. Feel free to try it, try your migration and send us your feedbacks. 

New and recent installations

If your instance has been initiated with Elasticsearch 6.x, you can follow the following process : 

  • Stop TheHive version 3.4.2
  • Stop Elasticsearch version 6.x
  • Update Elasticsearch configuration file
  • Update Elasticsearch to version 7.x and restart the service
  • Update TheHive and restart the service
  • Update Cortex and restart the service 

Instructions to install TheHive 3.5.0-RC1 or Cortex 3.1.0-RC1 can be found in this guide.

At this stage, connect TheHive and Cortex with your browser and you should be invited to update the database : 

Older indexes

This is the tricky part. If you are using an instance initiated with Elasticsearch older that version 6.0, it is highly probable that you have to follow an heavier process to upgrade. In few words, you will have to :  

  • Stop TheHive and Cortex applications
  • Create new indexes in Elasticsearch 6.x with part of the settings of your current indexes
  • Do specific reindexing operations to this new indexes
  • Delete old indexes.

How to identify if your index is ready for Elasticsearch 7

You can easily identify if indexes are ready for Elasticsearch 7. On the index named the_hive_15 run the following command:

curl -s http://127.0.0.1:9200/the_hive_15?human | jq '.the_hive_15.settings.index.version.created_string'

If the version is 6.x.x then the index will be read by Elasticsearch 7.8.x. Otherwise (version is 5.x.x of below), reindexing the index is required.

Migration guide

You are not left alone there. A dedicated documentation is available. It should help you run this specific actions on your Elasticsearch database, and also install or update application whether you are using DEB, RPM or binary packages, and even docker images : 

https://github.com/TheHive-Project/TheHiveDocs/blob/master/admin/upgrade_to_thehive_3_5_and_es_7_x.md

How to report issues

Please open an issue on GitHub using the template made for TheHive 3.x if you’d like to report a bug on this version. We will monitor those closely and respond accordingly. 

Cortex-Analyzers 2.9.0: Summer Edition!

Didn’t you think we were going to holidays without letting few new stuff to play with ? 6 new Analyzers and 1 Responder complete the growing list of Neurons.

A Huge thanks to all the contributors for the great new features, without forgetting the work regarding improvements and bug fixes.

Documentation

Find the complete documentation of Analyzers and Responders at the dedicated website: https://thehive-project.github.io/Cortex-Analyzers/

For each Analyzer and Responder, a page details the purpose of each flavor, the configuration required and even some screenshots from report samples.

New Analyzers

New Responder

Fixes and Improvements

  • Fix a bug in Hashdd_Detail_1_0 analyzer (#831)
  • Fix a bug in PhishingInitiative_Scan_1_0 analyzer (#832)
  • Fix a bug related to dataType in MalwareBazaar_1_0 analyzer (#830)
  • Fix a bug in MISPWarninglists analyzer (#827)
  • Fix a bug in Mailer_1_0 responder (#835)

Analyzers

DomainMailSPFDMARC

This analyzer comes in 1 flavor and let you check SPF (Sender Policy Framework) and DMARC (Domain-based Message Authentication, Reporting and Conformance) status of a domain or fqdn.

TheHive displays the analyzer results as follows:

DomainMailSPFDMARD short report
DomainMailSPFDMARD Long report

ForcepointWebsensePing

Forcepoint URL Filtering provides defenses against productivity draining web content and threats to operations. It ensures organizational productivity by delivering defenses against productivity draining web activity while providing the necessary security in a world of advanced threats.

Using WebsensePing utility is possible to query Master Database URL Categories that contains the industry’s most accurate, current and comprehensive classification of URLs. ForcePoint uses proprietary classification software and human inspection techniques to categorize and maintain definitions of more than 95 URL categories in more than 50 languages.

An active Forcepoint subscription is required to use the analyzer.

TheHive displays the analyzer results as follows:

ForcepointWebsensePing short report samples
ForcepointWebsensePing long report sample

NERD

This analyzer allows to query the NERD (Network Entity Reputation) database, and get score and basic information. Project NERD aims to build an extensive reputation database of known sources of cyber threats. That is, a list of known malicious IP addresses or other network entities (e.g. ASNs or domain names) together with all security-relevant information about each of them.

A valid API key is required to run this analyzer.

TheHive displays the analyzer results as follows:

NERD short report
NERD long report

SekoiaIntelligenceCenter

This analyzer allows you to gather more context related to domain names, IP adresses, urls and file hashes using the SEKOIA.IO Intelligence Database.

An active SEKOIA.IO Intelligence Center subscription is required to use the analyzer.

TheHive displays the analyzer results as follows:

SEKOIAIntelligenceCenter_Indicators long report

Spamassassin

This analyzer let you query a local SpamAssassin instance by sending a file, and get a SPAM score.

Apache SpamAssassin is the #1 Open Source anti-spam platform giving system administrators a filter to classify email and block spam (unsolicited bulk email). It uses a robust scoring framework and plug-ins to integrate a wide range of advanced heuristic and statistical analysis tests on email headers and body text including text analysis, Bayesian filtering, DNS blocklists, and collaborative filtering databases.

TheHive displays the analyzer results as follows:

Spamassassin short report
Spamassassin long report

Splunk

This analyzer allows you to execute a list of searches in Splunk by passing the element you are looking for as a parameter.

This analyzer comes in 10 flavors:

  • Splunk_Search_Domain_FQDN: Dispatch a list of saved searches on a given domain/fqdn
  • Splunk_Search_File_Filename: Dispatch a list of saved searches on a given file/filename
  • Splunk_Search_Hash: Dispatch a list of saved searches on a given hash
  • Splunk_Search_IP: Dispatch a list of saved searches on a given IP (IPv4 only)
  • Splunk_Search_Mail_Email: Dispatch a list of saved searches on a given mail/email
  • Splunk_Search_Mail_Subject: Dispatch a list of saved searches on a given mail_subject
  • Splunk_Search_Other: Dispatch a list of saved searches on a given data (any type)
  • Splunk_Search_Registry: Dispatch a list of saved searches on a given registry
  • Splunk_Search_URL_URI_Path: Dispatch a list of saved searches on a given url/uri_path
  • Splunk_Search_User_Agent: Dispatch a list of saved searches on a given user_agent
  • Splunk_Search_User: Dispatch a list of saved searches on a given user id (variable name is ‘other’)

A valid Splunk subscription is required to run this analyzer.

TheHive displays the analyzer results as follows:

Splunk_Search_Registry short report
Splunk_Search_Registry long report

Responders

Velociraptor

Velociraptor let you interrogate your endpoint for specific data. Velociraptor is a tool for collecting host based state information using Velocidex Query Language (VQL) queries.

This responder can be used to run a flow for a Velociraptor artifact.  This could include gathering data, or performing initial response.

It can be run on an observable type of ipfqdn, or other, and will look for a matching client via the Velociraptor server. If a client match is found for the last seen IP, or the hostname, the responder will kick off the flow, the results will be returned, and the client ID will be added as a tag to the case and the observable.

Get It While Supply Lasts!

If you are still using the old-style way of installing analyzers and responders, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done
for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu ( Admin > Analyzer templates in TheHive 4.0)
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

TheHive 4.0 is out!

Introduction

Several months,no, years ! after the first line of code – the first line was committed in 2016–, we are very excited and proud to announce the release of TheHive 4.0.

This means more than a major version for us. This was – and still is — like a completely new project, a new generation, a lot more challenging than before. We had to make the application climb a major step to introduce new key features, some we added in this version, others we have in mind for the future.

Objectives

The development of the second generation of TheHive, aka. version 4, was driven by three main objectives:

  • Add support to multi-tenancy: allow 1 instance of TheHive to serve many teams and organisations
  • Add support to Role Based Access Control to define fine grained user profiles
  • Rethink the data model and structure to support the goals listed above (Moving from Elasticsearch as main persistence layer, to a data model designed as a graph).

Challenges

TheHive Project is thoroughly adopted by SOC, CERTs and CSIRT teams, who decided to go with TheHive Project since the first releases. It is worth noting that until today, TheHive has had a total of 52 releases since 2016.

Those teams helped the project by contributing to our QA, questions, feature requests etc… and our way of thinking drove us to not let them down, and we decided to produce a backward compatible software.

The way we have been working until now aims to make our community move smoothly from TheHIve 3 to TheHive 4.

Backward compatibility

This is the most difficult challenge we have had, but we have hard heads and soft hearts.

TheHive 4 is expected to be backward compatible, thanks to APIs v0. Yes, we provide versioned APIs having the same endpoints as TheHive 3, and producing the same results. Search APIs also support the same query language, except some corner cases like searching using the “_string” operator (which is tightly coupled to Elasticsearch query language, but we have working alternatives).

Performance concerns

Supporting backward compatibility might force you to accept complex designs. And TheHive 4 RC3 was a clear example of that limitation.

Many kind users who tested TheHive 4 RC3, raised performance issues, slow UI problems etc… And it was completely expected. We thank them for making such a pressure on us, we used it to boost the refactoring of the UI, which was using backward compatible APIs (unoptimized for the new data model and representation), specially to read data (listing cases or observables for example).

We can discuss the technical details of this hard point later, but it mainly relates to navigating through graph-based data using a document based query system, which is not optimised.

For example, if you want to search for list of observable of a given case, the ideal way of doing that on a graph-base model is to:

  • Get the case by its ID, which is indexed (very fast operation)
  • Navigate through case relation, to find its links of type observable

But the backward compatible query language works differently: It scans all the graphs to search for observables that have a case parent with a given ID, which has a slower performance in a graph-based database.

Multi-tenancy and RBAC

TheHive 4 comes with a special multi-tenancy support. It allows the following strategies:

  • Use a siloed multi-tenancy: you can define many organisations, without allowing them to share data
  • Use a collaborative multi-tenancy: you can define a set of organizations and allow them to collaborate on specific cases/tasks/observables, using custom defined user profiles (RBAC)

This feature is very powerful but has a cost: an expected performance overhead. For example, when scanning the graph of data to search for a list of cases, TheHive must return the cases of your organisation and the case you can have access to because of the sharing rule.

New foundations

TheHive 3 was based on a framework called Elastic4play, written by Thomas to abstract all the routines required by a web application written with play 2 and using Elasticsearch.

TheHive 4 has its own core framework: Scalligraph, built to handle the following features.

Scalligraph will be the foundation of the next major version of Cortex.

What’s new in 4.0 

TheHive 4.0 release has a significant amount of changes. We will quickly explain the most important, and you can refer to the change logs if you need to have more details.

UI Performance

This was the most important task of this release. As we mentioned above, we were using backward compatible APIs in RC3 release, and migrated 80% of the UI to use the APIs v1 which are optimised for the new graph-based and multi-tenant data model.

OAuth2 Support

This topic gave birth to many github issues, some of them related to TheHive’s UI not correctly redirecting authenticated users. OAuth2 support has been tested with many providers like: Okta, Keycloak, FusionAuth, Microsoft Azure, Office 365 and Google Gsuite.

Starting from this version, there is an API endpoint that handle all the authentication and redirections: /api/ssoLogin

Here is a configuration sample for MS Office 365

{
  name: oauth2
  clientId: "CLIENT_ID"
  clientSecret: "CLIENT_SECRET"
  redirectUri: "http://THEHIVE_URL/api/ssoLogin"
  responseType: code
  grantType: "authorization_code"
  authorizationUrl: "https://login.microsoftonline.com/TENANT/oauth2/v2.0/authorize"
  authorizationHeader: "Bearer "
  tokenUrl: "https://login.microsoftonline.com/TENANT/oauth2/v2.0/token"
  userUrl: "https://graph.microsoft.com/v1.0/me"
  scope: ["User.Read"]
  userIdField: "mail" 
}

You can find more details about the OAuth2 support in the authentication config documentation

Improved Analyzer and Responder selection

Analyzer selection when calling bulk observable analysis has been improved to show the possible analyzers per observable type.

Analyzers selection during observable bulk analysis

For responders, the user experience has been improved as well, especially for instances with a big number of responders. The simple dropdown menu available to select responders has been replaced by a dialog allowing list filtering and scrolling:

New Responder selection dialog

Add bulk operations to case listing

Before this release, simple case updates required visiting the cases one by one and editing them. We added in this release a bulk edit feature, depending on user’s permissions on the selected cases

Bulk edit dialog, used here from case list

The same bulk editing component has been used to improve the same operations on observable list page.

Other noteworthy changes

We need to mention that the following changes have been included in TheHive 4.0 release:

  • Add pagination and filtering to users administration
  • Add back the UI configuration by organisation. The only available option is related to enabling/disabling the use of Empty Case.
  • Show sharing summary in task and observable lists
  • Improve alert preview dialog
  • Add alert externalLink feature allowing the display of external links for any alert, not only MISP alerts.

Known limitations

Even after 49 closed Github Issues, there are still major topics to be addressed by the upcoming releases:

  • Add back support to case merge which is not satisfying today. The challenge is to find the best to merge cases and make sure that it works in a profile-based multi-tenant design.
  • Add full text search support. In older versions, TheHive benefited from the full text search capabilities of Elasticsearch. With the new database and persistence system, full text support requires adding a dedicated indexing layer.

Installing and testing TheHive 4.0

After months of testing versions, this official release means that we consider it ready for production purposes. If you’re new with TheHive, we recommend going with TheHive 4.0.

Several installation guides have already been published, suitable with the chosen operating system and installation type, and new are coming.

For testing and training purposes, a virtual machine with a simple configuration of TheHive 4.0 and Cortex 3.0.1, is also published and available starting from now. Please refer to the documentation for download and usage instructions.

Want to upgrade from TheHive 3.x ?

All changes brought to TheHive make the upgrade more challenging than installing the new package and watch the progress bar. To support you with the upgrade, a migration tool comes along with the application to shift your current version of TheHive to TheHive 4.0.

A dedicated guide has been published to help users with this significant task. We recommend using a new server aside from your production server to ensure everything works fine with the migration.

Future of TheHive 3.x

This major outcome doesn’t mean TheHive 3 end of life is reached. As previously announced, we plan to support this version for some time, our next milestone being to support Elasticsearch 7.x with a first Release Candidate.

How to report issues

Please open an issue on GitHub using the template made for TheHive4 if you’d like to report a bug on this version. We will monitor those closely and respond accordingly.

Cortex-Analyzers 2.8.0: to infinity and beyond!

Thanks to the community and all the contributors, this release comes with 1 new Analyzer, 2 new Responders, lots of improvements and bug fixes.

But there is more news from the front.

Starting from this milestone, bugfixes and new Analyzers or Responders should be released in a smoother way as we are improving few processes. Some changes and recommandations should appear in the next days for submission, and our release process will be improved to fix bugs easier and release new code faster.

We also plan to offer a better documentation. We already started to publish information regarding each Analyzer and Responder. This is a work in progress, and it will be updated with the current requirements guide.

DomainToolsIris documentation page

For each Analyzer and Responder, a page details the purpose of each flavors, the configuration required and even some screenshots from report samples. It will also allow developers to share their own notes if wanted.

New Analyzers

New Responders

Improvements

  • Refactor Onyphe using new v2 api (#736)
  • Improvement in Shodan: add vulns in template and taxonomies (#772 & #776)
  • Improvement in Mailer responder: tasks support and auth (#764, #737, #379)
  • Improvement in SinkDb: support for new api with new dataTypes supported (#483, #498, #756)

Analyzers

LastInfoSec

LastInfoSec offers innovative and automated solutions to collect data, refine it and turn it into useful and actionable information, quickly available to improve the protection, detection and investigation capabilities of companies and government organizations.

TheHive displays the analyzer results as follows:

Short template for LastInfoSec Report
Long Template for LastInfoSec Report

Onyphe

An important work has been made on Onyphe Analyzer to support APIv2. All 7 flavors from older version have been removed and merged into only one flavor named “Onyphe_Summary”. An API key is still needed to query Onyphe API.

TheHive displays the analyzer results as follows:

Onyphe_Summary short report
Onyphe_Summary long report

Responders

Sendgrid

Sendgrid is a customer communication platform for transactional and marketing email used when you have to ensure that your notifications and transactional emails are delivered quickly and securely.

This analyzer works like the Mailer one, but relying on SendGrid external service to delivery emails.

In order to use the service please follow the instruction being careful to the verify your email address.

VirusTotalDownloader

This responders runs on Observables of type “hash” and allows analyst to download corresponding file from VirusTotal. Once downloaded, the file is added to the current case observables in TheHive.

In order to use this responder, a Premium API key from VirusTotal is needed. An API key from TheHive is also needed to upload the file in the observables list.

Use the responder on the hash to add the sample in your Observables

Fixes and Improvements

  • Fix: some analyzer uses invalid “email” dataType (#799)
  • Fix in MalwareBazaar: wrong dataTypes in config (#794)
  • Fix in PhishTank: the JSON object must be str, not ‘bytes’ (#786)
  • Fix in VMRay: fix error in parsing and workflow (#785 & #784)
  • Fix in Wazuh: ipaddress import missing (#778)
  • Fix in Minemeld Responder: requests missing in requirements (#774)
  • Fix in WOT: moving to new endpoint (#771)
  • DomainTools Iris: update api urls (#760)
  • Fix in ThreatResponse: module_type key removed from response (#759)
  • Fix in Abuse_Finder: pythonwhois dependency (#742)

Get It While Supply Lasts!

If you are still using the old-style way of installing analyzers and responders, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done
for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates` menu
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

TheHive4py got a second wind, version 1.7.0 released

“TheHive4py”, this sounds like a word you didn’t hear about during the last 12 months. Well, our focus on this library was put on hold. We will tell you the reason, but much better, we will solve the problem.

A brief review

TheHive4py was quickly initiated after the first releases of TheHive to help developers interact with TheHive APIs using python. We started creating methods and functions for main functionalities and to be honest, it was a sort of a quick-and-dirty solution.

TheHive4py has some limitation:

  • The API client is a flat class with dozens of methods
  • The API clients’ methods return the native `requests.Reponse` class instead of a structured data
  • Exception handling could be improved
  • Code could be made more reusable

As developers, we are aware of these limitations and are eager to provide a better library, and that’s what we started making with TheHive4py rewrite. We wanted to provide you with a library you can use this way:

# Fetch cases
open_cases = api.cases.find_all({'status': 'Open'}, range='0-5')
log('Open cases', list(map(lambda i: i.json(), open_cases)))

# Fetch a case by `id` or `number` (caseId)
sample_case = open_cases[0]
log('case details by id', api.cases.get_by_id(sample_case.id).json())
log('case details by number', api.cases.get_by_number(sample_case.caseId).json())

# Fetch alerts
new_alerts = api.alerts.find_all({'status': 'New'}, range='0-2')
log('New alerts', list(map(lambda i: i.json(), new_alerts)))

# Fetch observables
domain_observables = api.observables.find_all({'dataType': 'domain'}, range='0-2')
log('New alerts', list(map(lambda i: i.json(), domain_observables)))

# Fetch tasks
waiting_tasks = api.tasks.find_all({'status': 'Waiting'}, range='0-2')
log('Waiting tasks', list(map(lambda i: i.json(), waiting_tasks)))

waiting_tasks = api.tasks.get_waiting(range='0-2')
log('Waiting tasks', list(map(lambda i: i.json(), waiting_tasks)))

jdoe_tasks = api.tasks.get_by_user('jdoe', {}, range='0-3')
log('Tasks of jdoe', list(map(lambda i: i.json(), jdoe_tasks)))

case_tasks = api.tasks.of_case(sample_case.id, query={'status': 'Waiting'})
log('Case tasks', list(map(lambda i: i.json(), case_tasks)))

The library’s rewrite was supposed to produce a 2.0.0 version of TheHive4py but we had a major issue: backward compatibility.

Well, in theory, backward compatibility can be handled through a clear communication to:

  • tell the users how to make sure to update their dependencies to TheHive4py < 2.0.0
  • provide a migration plan
  • maintain both versions during a certain time
  • maintain documentation for old and new versions

To be honest, this was hard to achieve, because of the famous lack of time, but things a going to change.

What’s the plan?

We didn’t want to make a plan without asking the community about how they interact with TheHive APIs. So we did two twitter polls that ended up with the following results:

Twitter poll about TheHive API usage methods

The second poll asked our users about pros and cons of TheHive4py:

Twitter poll about TheHive4py pros and cons

The poll results are clear: we need to put more efforts on TheHive4py.

Here we go, firstly, let’s release version 1.7.0

TheHive4py 1.7.0 milestone has been initiated almost one year ago, and we are happy to announce its availability today.

What’s new about it?

The most important change is allowing TheHive4py to interact with TheHive 4 in addition to introducing some missing features, and bug fixes. Here is a short listing of main changes:

Add support to multi tenancy

Allow a developer to specify the organisation against which an API call is done:

api = TheHiveApi('http://my_thehive:9000', 'my_api_key', organisation='cert')

Add custom field support for new types:

TheHive 4 introduces custom fields of type integer and float, this feature allows specifying custom fields with types supported by TheHive 4. These types are not supported by TheHive 3.

CustomFieldHelper
   .add_integer('number_hits', 10)
   .add_float('cvss', 5.6)
   .build()

The code snippet above produces the following content:

{
  "number_hits": {
    "order": 0,
    "integer": 100
  },
  "cvss": {
    "order": 1,
    "integer": 5.6
  }
}

Add support to like and wildcard query operators

TheHive query DSL supports like and wildcard operators, but TheHive4py didn’t had an option to use those operators. In this version the following query methods have been added:

  • Like (field, value): Field’s value must contain value, that must contain `*` in the beginning or at the end
  • StratsWith (field, value): Field’s value must start with value
  • EndsWith (field, value): Field’s value must end with value
  • ContainsString (field, value): Field’s value must contain value
from thehive4py.query import Eq, Like, And, StartsWith

# find cases where title contains 'Dridex'
api.find_cases(query=Like('title', 'Dridex*'))

# find alerts where status is 'New' and title starts with 'Emotet'
api.find_alert(query=And(Eq('status', 'New'), StartsWith('title', 'Emotet')))

Add ioc and sighted attributes to case and alert artifacts

This allows specifying these attributes during Alert or Case observables creation

Add update_case_observable method

Can be used to patch an existing observable, by setting a tag or marking as IOC.

Add PAP to Case and CaseTemplate models

PAP flag has been added in TheHive recently and TheHive4py was not able to set the PAP value of a Case or CaseTemplate

Add custom fields creation method

Added a `create_custom_field` method that check custom field name uniqueness before creating it.

Note: This method is for now, compatible with TheHive 3 only because it relies on the DBList API that is no longer available on TheHive 4.

Add case template creation method

Added a `create_case_template` method allowing developers to create new Case Templates.

The full change log is available at the release page

What about documentation

Once again we are glad to announce the initial version of a documentation website, dedicated to TheHive4py, including documentation of all the features the library provides, and code samples of the most useful features.

We aim to maintain and improve this documentation over time, so please, don’t hesitate to either contribute or ask for more content.

Screenshot of the documentation website

TheHive4py 2.0

We will put the rewrite of TheHive4py on hold for now and will communicate about it again when we are ready. In the meantime, we will continue maintaining TheHive4py 1.x.

Update: TheHive4py 1.7.1 Patch

During the release 1.7.0, we have noticed that the build process and deployment went wrong, so we have created a 1.7.0.post1 release.

The community also raised a regression that has been fixed in 1.7.1 release. You can read the change log for more details.

Updating/Installing

To update your existing package to version 1.7.0:

$ sudo pip install thehive4py --upgrade

Got a question?

If you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. As usual, we’ll be more than happy to help!

ElasticSearch, TheHive and Cortex

For a few weeks, many questions have been arising regarding the End of Life of ElasticSearch 6.8, and its impact on TheHive and Cortex applications.

We were about to release TheHive 4.0-RC3 when Thomas, akwardly calmly announced to us, having found some time (where?) to review new features and most important, breaking changes introduced by ES7. We have now a good idea of what should be updated or added in the code, as well as the amont of work it represents to get the application working perfectly.

What about current version ?

Few months ago, we announced our intention to maintain current stable versions until ES6 End of Life. At that time, we didn’t expect it to be sooner.

Discontinuing TheHive 3.x with the release of TheHive 4.0 has never been in our plans. With the time, more and more organisations adopted them, and it is important for us to give everyone enough space to schedule and make the move to the TheHive 4.0. This is why TheHive 3 and Cortex 3 will support ES7.

The good news is our ability to announce that the changes introduced by ES7 have no major impacts on us, We are scheduling a first RC1 for TheHive 3.5.0 and Cortex 3.1.0 in the last week of July. Not only will they include support for ES7, but also a few interesting improvements that will be introduced in the coming blog posts.

What’s next ?

Needless to say, the chiefs are sparing no effort in focusing on TheHive 4.0, which requires a huge amount of attention. The application stack has completely changed – the most important adjustment is pushing aside ElasticSearch in favour of Cassandra to manage TheHive’s data storage – and thanks to the community, lots of bugs have already been fixed allowing it to be stronger with time.

Once we consider TheHive 4.0 reliable enough to be used in production, we will publish it as a stable version, and that would be in the coming days. After all, our plans are to make the applications use the same technology stack, which will directly benefit to the next major version of Cortex.

Besides, Cortex is scheduled to be upgraded and based upon Scalligraph, Cassandra and Hadoop. We hope to publish a first RC in few months.

Stay tuned sounds like TheHive Project’s Twitter account will be on fire 🔥 in the coming days!