TheHive 3.5.0-RC1 and Cortex 3.1.0-RC1 are here!

Elasticsearch 7.0 can read indices created in version 6.0 or above. An Elasticsearch 7.0 node will not start in the presence of indices created in a version of Elasticsearch before 6.0.

https://www.elastic.co/guide/en/elasticsearch/reference/7.x/breaking-changes-7.0.html#_indices_created_before_7_0

Who could imagine what’s hiding behind this sentence ? 

To be honest, we managed to support Elasticsearch 7.x pretty quickly ! But only for new and recent installations and instances — read, initially installed with Elasticsearch 6.x. 

The harder part was ensure older instances, with indexes created  with Elasticsearch 5.x, can migrate smoothly like for previous migrations: «stop the application, update the database software, update application and restart everything ». You might have to put your hands on the keyboard.

Source: Google Images

⚠️  TheHive 3.5.0-RC1 and Cortex 3.1.0-RC1 are not recommended for production use. These versions are intended for test only ; please, read carefully the full blog post and the associated documentation. Feel free to try it, try your migration and send us your feedbacks. 

New and recent installations

If your instance has been initiated with Elasticsearch 6.x, you can follow the following process : 

  • Stop TheHive version 3.4.2
  • Stop Elasticsearch version 6.x
  • Update Elasticsearch configuration file
  • Update Elasticsearch to version 7.x and restart the service
  • Update TheHive and restart the service
  • Update Cortex and restart the service 

Instructions to install TheHive 3.5.0-RC1 or Cortex 3.1.0-RC1 can be found in this guide.

At this stage, connect TheHive and Cortex with your browser and you should be invited to update the database : 

Older indexes

This is the tricky part. If you are using an instance initiated with Elasticsearch older that version 6.0, it is highly probable that you have to follow an heavier process to upgrade. In few words, you will have to :  

  • Stop TheHive and Cortex applications
  • Create new indexes in Elasticsearch 6.x with part of the settings of your current indexes
  • Do specific reindexing operations to this new indexes
  • Delete old indexes.

How to identify if your index is ready for Elasticsearch 7

You can easily identify if indexes are ready for Elasticsearch 7. On the index named the_hive_15 run the following command:

curl -s http://127.0.0.1:9200/the_hive_15?human | jq '.the_hive_15.settings.index.version.created_string'

If the version is 6.x.x then the index will be read by Elasticsearch 7.8.x. Otherwise (version is 5.x.x of below), reindexing the index is required.

Migration guide

You are not left alone there. A dedicated documentation is available. It should help you run this specific actions on your Elasticsearch database, and also install or update application whether you are using DEB, RPM or binary packages, and even docker images : 

https://github.com/TheHive-Project/TheHiveDocs/blob/master/admin/upgrade_to_thehive_3_5_and_es_7_x.md

How to report issues

Please open an issue on GitHub using the template made for TheHive 3.x if you’d like to report a bug on this version. We will monitor those closely and respond accordingly. 

Cortex-Analyzers 2.9.0: Summer Edition!

Didn’t you think we were going to holidays without letting few new stuff to play with ? 6 new Analyzers and 1 Responder complete the growing list of Neurons.

A Huge thanks to all the contributors for the great new features, without forgetting the work regarding improvements and bug fixes.

Documentation

Find the complete documentation of Analyzers and Responders at the dedicated website: https://thehive-project.github.io/Cortex-Analyzers/

For each Analyzer and Responder, a page details the purpose of each flavor, the configuration required and even some screenshots from report samples.

New Analyzers

New Responder

Fixes and Improvements

  • Fix a bug in Hashdd_Detail_1_0 analyzer (#831)
  • Fix a bug in PhishingInitiative_Scan_1_0 analyzer (#832)
  • Fix a bug related to dataType in MalwareBazaar_1_0 analyzer (#830)
  • Fix a bug in MISPWarninglists analyzer (#827)
  • Fix a bug in Mailer_1_0 responder (#835)

Analyzers

DomainMailSPFDMARC

This analyzer comes in 1 flavor and let you check SPF (Sender Policy Framework) and DMARC (Domain-based Message Authentication, Reporting and Conformance) status of a domain or fqdn.

TheHive displays the analyzer results as follows:

DomainMailSPFDMARD short report
DomainMailSPFDMARD Long report

ForcepointWebsensePing

Forcepoint URL Filtering provides defenses against productivity draining web content and threats to operations. It ensures organizational productivity by delivering defenses against productivity draining web activity while providing the necessary security in a world of advanced threats.

Using WebsensePing utility is possible to query Master Database URL Categories that contains the industry’s most accurate, current and comprehensive classification of URLs. ForcePoint uses proprietary classification software and human inspection techniques to categorize and maintain definitions of more than 95 URL categories in more than 50 languages.

An active Forcepoint subscription is required to use the analyzer.

TheHive displays the analyzer results as follows:

ForcepointWebsensePing short report samples
ForcepointWebsensePing long report sample

NERD

This analyzer allows to query the NERD (Network Entity Reputation) database, and get score and basic information. Project NERD aims to build an extensive reputation database of known sources of cyber threats. That is, a list of known malicious IP addresses or other network entities (e.g. ASNs or domain names) together with all security-relevant information about each of them.

A valid API key is required to run this analyzer.

TheHive displays the analyzer results as follows:

NERD short report
NERD long report

SekoiaIntelligenceCenter

This analyzer allows you to gather more context related to domain names, IP adresses, urls and file hashes using the SEKOIA.IO Intelligence Database.

An active SEKOIA.IO Intelligence Center subscription is required to use the analyzer.

TheHive displays the analyzer results as follows:

SEKOIAIntelligenceCenter_Indicators long report

Spamassassin

This analyzer let you query a local SpamAssassin instance by sending a file, and get a SPAM score.

Apache SpamAssassin is the #1 Open Source anti-spam platform giving system administrators a filter to classify email and block spam (unsolicited bulk email). It uses a robust scoring framework and plug-ins to integrate a wide range of advanced heuristic and statistical analysis tests on email headers and body text including text analysis, Bayesian filtering, DNS blocklists, and collaborative filtering databases.

TheHive displays the analyzer results as follows:

Spamassassin short report
Spamassassin long report

Splunk

This analyzer allows you to execute a list of searches in Splunk by passing the element you are looking for as a parameter.

This analyzer comes in 10 flavors:

  • Splunk_Search_Domain_FQDN: Dispatch a list of saved searches on a given domain/fqdn
  • Splunk_Search_File_Filename: Dispatch a list of saved searches on a given file/filename
  • Splunk_Search_Hash: Dispatch a list of saved searches on a given hash
  • Splunk_Search_IP: Dispatch a list of saved searches on a given IP (IPv4 only)
  • Splunk_Search_Mail_Email: Dispatch a list of saved searches on a given mail/email
  • Splunk_Search_Mail_Subject: Dispatch a list of saved searches on a given mail_subject
  • Splunk_Search_Other: Dispatch a list of saved searches on a given data (any type)
  • Splunk_Search_Registry: Dispatch a list of saved searches on a given registry
  • Splunk_Search_URL_URI_Path: Dispatch a list of saved searches on a given url/uri_path
  • Splunk_Search_User_Agent: Dispatch a list of saved searches on a given user_agent
  • Splunk_Search_User: Dispatch a list of saved searches on a given user id (variable name is ‘other’)

A valid Splunk subscription is required to run this analyzer.

TheHive displays the analyzer results as follows:

Splunk_Search_Registry short report
Splunk_Search_Registry long report

Responders

Velociraptor

Velociraptor let you interrogate your endpoint for specific data. Velociraptor is a tool for collecting host based state information using Velocidex Query Language (VQL) queries.

This responder can be used to run a flow for a Velociraptor artifact.  This could include gathering data, or performing initial response.

It can be run on an observable type of ipfqdn, or other, and will look for a matching client via the Velociraptor server. If a client match is found for the last seen IP, or the hostname, the responder will kick off the flow, the results will be returned, and the client ID will be added as a tag to the case and the observable.

Get It While Supply Lasts!

If you are still using the old-style way of installing analyzers and responders, run the following commands:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done
for I in responders/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

Once done, ensure to refresh your analyzers and responders in the Cortex WebUI. Connect as an orgadmin and go to the Organization menu. Click on the Analyzers tab and click on the Refresh analyzers button. Do the same for the Responders tab: click on the Refresh responders button. Refer to the online Cortex documentation for further details.

Update TheHive Report Templates

If you are using TheHive, you must import the new report templates in your instance as follows:

  • download the updated package
  • log in TheHive using an administrator account
  • go to Admin > Report templates menu ( Admin > Analyzer templates in TheHive 4.0)
  • click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

TheHive 4.0 is out!

Introduction

Several months,no, years ! after the first line of code – the first line was committed in 2016–, we are very excited and proud to announce the release of TheHive 4.0.

This means more than a major version for us. This was – and still is — like a completely new project, a new generation, a lot more challenging than before. We had to make the application climb a major step to introduce new key features, some we added in this version, others we have in mind for the future.

Objectives

The development of the second generation of TheHive, aka. version 4, was driven by three main objectives:

  • Add support to multi-tenancy: allow 1 instance of TheHive to serve many teams and organisations
  • Add support to Role Based Access Control to define fine grained user profiles
  • Rethink the data model and structure to support the goals listed above (Moving from Elasticsearch as main persistence layer, to a data model designed as a graph).

Challenges

TheHive Project is thoroughly adopted by SOC, CERTs and CSIRT teams, who decided to go with TheHive Project since the first releases. It is worth noting that until today, TheHive has had a total of 52 releases since 2016.

Those teams helped the project by contributing to our QA, questions, feature requests etc… and our way of thinking drove us to not let them down, and we decided to produce a backward compatible software.

The way we have been working until now aims to make our community move smoothly from TheHIve 3 to TheHive 4.

Backward compatibility

This is the most difficult challenge we have had, but we have hard heads and soft hearts.

TheHive 4 is expected to be backward compatible, thanks to APIs v0. Yes, we provide versioned APIs having the same endpoints as TheHive 3, and producing the same results. Search APIs also support the same query language, except some corner cases like searching using the “_string” operator (which is tightly coupled to Elasticsearch query language, but we have working alternatives).

Performance concerns

Supporting backward compatibility might force you to accept complex designs. And TheHive 4 RC3 was a clear example of that limitation.

Many kind users who tested TheHive 4 RC3, raised performance issues, slow UI problems etc… And it was completely expected. We thank them for making such a pressure on us, we used it to boost the refactoring of the UI, which was using backward compatible APIs (unoptimized for the new data model and representation), specially to read data (listing cases or observables for example).

We can discuss the technical details of this hard point later, but it mainly relates to navigating through graph-based data using a document based query system, which is not optimised.

For example, if you want to search for list of observable of a given case, the ideal way of doing that on a graph-base model is to:

  • Get the case by its ID, which is indexed (very fast operation)
  • Navigate through case relation, to find its links of type observable

But the backward compatible query language works differently: It scans all the graphs to search for observables that have a case parent with a given ID, which has a slower performance in a graph-based database.

Multi-tenancy and RBAC

TheHive 4 comes with a special multi-tenancy support. It allows the following strategies:

  • Use a siloed multi-tenancy: you can define many organisations, without allowing them to share data
  • Use a collaborative multi-tenancy: you can define a set of organizations and allow them to collaborate on specific cases/tasks/observables, using custom defined user profiles (RBAC)

This feature is very powerful but has a cost: an expected performance overhead. For example, when scanning the graph of data to search for a list of cases, TheHive must return the cases of your organisation and the case you can have access to because of the sharing rule.

New foundations

TheHive 3 was based on a framework called Elastic4play, written by Thomas to abstract all the routines required by a web application written with play 2 and using Elasticsearch.

TheHive 4 has its own core framework: Scalligraph, built to handle the following features.

Scalligraph will be the foundation of the next major version of Cortex.

What’s new in 4.0 

TheHive 4.0 release has a significant amount of changes. We will quickly explain the most important, and you can refer to the change logs if you need to have more details.

UI Performance

This was the most important task of this release. As we mentioned above, we were using backward compatible APIs in RC3 release, and migrated 80% of the UI to use the APIs v1 which are optimised for the new graph-based and multi-tenant data model.

OAuth2 Support

This topic gave birth to many github issues, some of them related to TheHive’s UI not correctly redirecting authenticated users. OAuth2 support has been tested with many providers like: Okta, Keycloak, FusionAuth, Microsoft Azure, Office 365 and Google Gsuite.

Starting from this version, there is an API endpoint that handle all the authentication and redirections: /api/ssoLogin

Here is a configuration sample for MS Office 365

{
  name: oauth2
  clientId: "CLIENT_ID"
  clientSecret: "CLIENT_SECRET"
  redirectUri: "http://THEHIVE_URL/api/ssoLogin"
  responseType: code
  grantType: "authorization_code"
  authorizationUrl: "https://login.microsoftonline.com/TENANT/oauth2/v2.0/authorize"
  authorizationHeader: "Bearer "
  tokenUrl: "https://login.microsoftonline.com/TENANT/oauth2/v2.0/token"
  userUrl: "https://graph.microsoft.com/v1.0/me"
  scope: ["User.Read"]
  userIdField: "mail" 
}

You can find more details about the OAuth2 support in the authentication config documentation

Improved Analyzer and Responder selection

Analyzer selection when calling bulk observable analysis has been improved to show the possible analyzers per observable type.

Analyzers selection during observable bulk analysis

For responders, the user experience has been improved as well, especially for instances with a big number of responders. The simple dropdown menu available to select responders has been replaced by a dialog allowing list filtering and scrolling:

New Responder selection dialog

Add bulk operations to case listing

Before this release, simple case updates required visiting the cases one by one and editing them. We added in this release a bulk edit feature, depending on user’s permissions on the selected cases

Bulk edit dialog, used here from case list

The same bulk editing component has been used to improve the same operations on observable list page.

Other noteworthy changes

We need to mention that the following changes have been included in TheHive 4.0 release:

  • Add pagination and filtering to users administration
  • Add back the UI configuration by organisation. The only available option is related to enabling/disabling the use of Empty Case.
  • Show sharing summary in task and observable lists
  • Improve alert preview dialog
  • Add alert externalLink feature allowing the display of external links for any alert, not only MISP alerts.

Known limitations

Even after 49 closed Github Issues, there are still major topics to be addressed by the upcoming releases:

  • Add back support to case merge which is not satisfying today. The challenge is to find the best to merge cases and make sure that it works in a profile-based multi-tenant design.
  • Add full text search support. In older versions, TheHive benefited from the full text search capabilities of Elasticsearch. With the new database and persistence system, full text support requires adding a dedicated indexing layer.

Installing and testing TheHive 4.0

After months of testing versions, this official release means that we consider it ready for production purposes. If you’re new with TheHive, we recommend going with TheHive 4.0.

Several installation guides have already been published, suitable with the chosen operating system and installation type, and new are coming.

For testing and training purposes, a virtual machine with a simple configuration of TheHive 4.0 and Cortex 3.0.1, is also published and available starting from now. Please refer to the documentation for download and usage instructions.

Want to upgrade from TheHive 3.x ?

All changes brought to TheHive make the upgrade more challenging than installing the new package and watch the progress bar. To support you with the upgrade, a migration tool comes along with the application to shift your current version of TheHive to TheHive 4.0.

A dedicated guide has been published to help users with this significant task. We recommend using a new server aside from your production server to ensure everything works fine with the migration.

Future of TheHive 3.x

This major outcome doesn’t mean TheHive 3 end of life is reached. As previously announced, we plan to support this version for some time, our next milestone being to support Elasticsearch 7.x with a first Release Candidate.

How to report issues

Please open an issue on GitHub using the template made for TheHive4 if you’d like to report a bug on this version. We will monitor those closely and respond accordingly.

TheHive4py got a second wind, version 1.7.0 released

“TheHive4py”, this sounds like a word you didn’t hear about during the last 12 months. Well, our focus on this library was put on hold. We will tell you the reason, but much better, we will solve the problem.

A brief review

TheHive4py was quickly initiated after the first releases of TheHive to help developers interact with TheHive APIs using python. We started creating methods and functions for main functionalities and to be honest, it was a sort of a quick-and-dirty solution.

TheHive4py has some limitation:

  • The API client is a flat class with dozens of methods
  • The API clients’ methods return the native `requests.Reponse` class instead of a structured data
  • Exception handling could be improved
  • Code could be made more reusable

As developers, we are aware of these limitations and are eager to provide a better library, and that’s what we started making with TheHive4py rewrite. We wanted to provide you with a library you can use this way:

# Fetch cases
open_cases = api.cases.find_all({'status': 'Open'}, range='0-5')
log('Open cases', list(map(lambda i: i.json(), open_cases)))

# Fetch a case by `id` or `number` (caseId)
sample_case = open_cases[0]
log('case details by id', api.cases.get_by_id(sample_case.id).json())
log('case details by number', api.cases.get_by_number(sample_case.caseId).json())

# Fetch alerts
new_alerts = api.alerts.find_all({'status': 'New'}, range='0-2')
log('New alerts', list(map(lambda i: i.json(), new_alerts)))

# Fetch observables
domain_observables = api.observables.find_all({'dataType': 'domain'}, range='0-2')
log('New alerts', list(map(lambda i: i.json(), domain_observables)))

# Fetch tasks
waiting_tasks = api.tasks.find_all({'status': 'Waiting'}, range='0-2')
log('Waiting tasks', list(map(lambda i: i.json(), waiting_tasks)))

waiting_tasks = api.tasks.get_waiting(range='0-2')
log('Waiting tasks', list(map(lambda i: i.json(), waiting_tasks)))

jdoe_tasks = api.tasks.get_by_user('jdoe', {}, range='0-3')
log('Tasks of jdoe', list(map(lambda i: i.json(), jdoe_tasks)))

case_tasks = api.tasks.of_case(sample_case.id, query={'status': 'Waiting'})
log('Case tasks', list(map(lambda i: i.json(), case_tasks)))

The library’s rewrite was supposed to produce a 2.0.0 version of TheHive4py but we had a major issue: backward compatibility.

Well, in theory, backward compatibility can be handled through a clear communication to:

  • tell the users how to make sure to update their dependencies to TheHive4py < 2.0.0
  • provide a migration plan
  • maintain both versions during a certain time
  • maintain documentation for old and new versions

To be honest, this was hard to achieve, because of the famous lack of time, but things a going to change.

What’s the plan?

We didn’t want to make a plan without asking the community about how they interact with TheHive APIs. So we did two twitter polls that ended up with the following results:

Twitter poll about TheHive API usage methods

The second poll asked our users about pros and cons of TheHive4py:

Twitter poll about TheHive4py pros and cons

The poll results are clear: we need to put more efforts on TheHive4py.

Here we go, firstly, let’s release version 1.7.0

TheHive4py 1.7.0 milestone has been initiated almost one year ago, and we are happy to announce its availability today.

What’s new about it?

The most important change is allowing TheHive4py to interact with TheHive 4 in addition to introducing some missing features, and bug fixes. Here is a short listing of main changes:

Add support to multi tenancy

Allow a developer to specify the organisation against which an API call is done:

api = TheHiveApi('http://my_thehive:9000', 'my_api_key', organisation='cert')

Add custom field support for new types:

TheHive 4 introduces custom fields of type integer and float, this feature allows specifying custom fields with types supported by TheHive 4. These types are not supported by TheHive 3.

CustomFieldHelper
   .add_integer('number_hits', 10)
   .add_float('cvss', 5.6)
   .build()

The code snippet above produces the following content:

{
  "number_hits": {
    "order": 0,
    "integer": 100
  },
  "cvss": {
    "order": 1,
    "integer": 5.6
  }
}

Add support to like and wildcard query operators

TheHive query DSL supports like and wildcard operators, but TheHive4py didn’t had an option to use those operators. In this version the following query methods have been added:

  • Like (field, value): Field’s value must contain value, that must contain `*` in the beginning or at the end
  • StratsWith (field, value): Field’s value must start with value
  • EndsWith (field, value): Field’s value must end with value
  • ContainsString (field, value): Field’s value must contain value
from thehive4py.query import Eq, Like, And, StartsWith

# find cases where title contains 'Dridex'
api.find_cases(query=Like('title', 'Dridex*'))

# find alerts where status is 'New' and title starts with 'Emotet'
api.find_alert(query=And(Eq('status', 'New'), StartsWith('title', 'Emotet')))

Add ioc and sighted attributes to case and alert artifacts

This allows specifying these attributes during Alert or Case observables creation

Add update_case_observable method

Can be used to patch an existing observable, by setting a tag or marking as IOC.

Add PAP to Case and CaseTemplate models

PAP flag has been added in TheHive recently and TheHive4py was not able to set the PAP value of a Case or CaseTemplate

Add custom fields creation method

Added a `create_custom_field` method that check custom field name uniqueness before creating it.

Note: This method is for now, compatible with TheHive 3 only because it relies on the DBList API that is no longer available on TheHive 4.

Add case template creation method

Added a `create_case_template` method allowing developers to create new Case Templates.

The full change log is available at the release page

What about documentation

Once again we are glad to announce the initial version of a documentation website, dedicated to TheHive4py, including documentation of all the features the library provides, and code samples of the most useful features.

We aim to maintain and improve this documentation over time, so please, don’t hesitate to either contribute or ask for more content.

Screenshot of the documentation website

TheHive4py 2.0

We will put the rewrite of TheHive4py on hold for now and will communicate about it again when we are ready. In the meantime, we will continue maintaining TheHive4py 1.x.

Update: TheHive4py 1.7.1 Patch

During the release 1.7.0, we have noticed that the build process and deployment went wrong, so we have created a 1.7.0.post1 release.

The community also raised a regression that has been fixed in 1.7.1 release. You can read the change log for more details.

Updating/Installing

To update your existing package to version 1.7.0:

$ sudo pip install thehive4py --upgrade

Got a question?

If you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. As usual, we’ll be more than happy to help!

ElasticSearch, TheHive and Cortex

For a few weeks, many questions have been arising regarding the End of Life of ElasticSearch 6.8, and its impact on TheHive and Cortex applications.

We were about to release TheHive 4.0-RC3 when Thomas, akwardly calmly announced to us, having found some time (where?) to review new features and most important, breaking changes introduced by ES7. We have now a good idea of what should be updated or added in the code, as well as the amont of work it represents to get the application working perfectly.

What about current version ?

Few months ago, we announced our intention to maintain current stable versions until ES6 End of Life. At that time, we didn’t expect it to be sooner.

Discontinuing TheHive 3.x with the release of TheHive 4.0 has never been in our plans. With the time, more and more organisations adopted them, and it is important for us to give everyone enough space to schedule and make the move to the TheHive 4.0. This is why TheHive 3 and Cortex 3 will support ES7.

The good news is our ability to announce that the changes introduced by ES7 have no major impacts on us, We are scheduling a first RC1 for TheHive 3.5.0 and Cortex 3.1.0 in the last week of July. Not only will they include support for ES7, but also a few interesting improvements that will be introduced in the coming blog posts.

What’s next ?

Needless to say, the chiefs are sparing no effort in focusing on TheHive 4.0, which requires a huge amount of attention. The application stack has completely changed – the most important adjustment is pushing aside ElasticSearch in favour of Cassandra to manage TheHive’s data storage – and thanks to the community, lots of bugs have already been fixed allowing it to be stronger with time.

Once we consider TheHive 4.0 reliable enough to be used in production, we will publish it as a stable version, and that would be in the coming days. After all, our plans are to make the applications use the same technology stack, which will directly benefit to the next major version of Cortex.

Besides, Cortex is scheduled to be upgraded and based upon Scalligraph, Cassandra and Hadoop. We hope to publish a first RC in few months.

Stay tuned sounds like TheHive Project’s Twitter account will be on fire 🔥 in the coming days!

TheHive 4.0-RC3, a new kid on the block

Three weeks ago, on May 6th, we announced the availability of TheHive 4.0 RC-2 release and the active community made the usual hard work of testing the release to find bugs and ask for enhancements.

Special thanks to Christopher, alias crackytsi who has already created 122 Github issues, 11 of them are just for 4.0-RC-3 milestone.

We are thrilled to present our third release candidate before the final release of TheHive 4. As the RC-2, this milestone brings new features and fixes a nice set of issues.

This blog post will focus on the following new features:

  • Multi-orgs users
  • Switch organisation
  • OAuth2
  • Migration tool performance
  • Case sharing overview

If you are curious about all the issues that have been addressed, you can read the full changelog

Multi-orgs users

This feature doesn’t introduce UI changes, but it allows a superadmin or an org-admin to add an existing user to an organisation.

Users in TheHive 4 are identified by their email addresses, so when an administrator adds a new user, with an email address that already exists, TheHive 4 links that existing user to the organisation being updated.

This ends up with a single User record on the database, linked to multiple organisations. Thanks to the new graph data model. This means the given user has:

  • the same credentials
  • the same api key, if enabled
  • the same 2FA settings, if enables
  • the same status (locked or not)

With that being said, the user can have a different profile for on the organisation (s)he belongs to.

What happens when a user is logged in?

As we mentioned earlier, a user belonging to several organisations, has the same authentication settings, and after the login, his/her workspace is opened with the context of the first organisation (s)he has been created on.

For example, if John was firstly created on the *SocLevel2* organisation, and was later attached to *CTI* organisation, then after signing in, the user is redirected to the workspace of *SocLevel2* organisation.

Future improvements

We will consider allowing the user to define a default organisation to be displayed juste after the login. We are examining the possibility to allow the user to define a default organisation to be displayed just after the login process. Hopefully, we will be able to add this feature in TheHive 4.0.0 release.

Switch organisation

This feature empowers the multi-tenancy capabilities brought to you by TheHive 4. Following what has been showcased above, how can a user, who belongs to more than one organisation, switch between his/her tenants?

The UI introduces a simple feature, available to “multi-org” users only, as a button on the right hand side of the page’s header, aka. the navigation bar.

The switch organisation action button

This button is just hidden for users who belong to a single organisation.

Once clicked, that button show a dialog that displays the following details:

  • user’s organisations
  • user’s profile on each organisation
  • the current organisation

Clicking on an item of this list, refreshes the page by loading the context of the selected organisation, and the UI behaves like if the user was logged in a a member of that selected organisation.

Very useful.

Switch organisation dialog

OAuth2

We had a considerable amount of users asking for SSO and OAuth support in TheHive. We tried to make it more robust in TheHive 4, and let it rely on a redirectUri provided by the backend (/api/ssoLogin) instead of the old redirectUri that some OAuth providers don’t support (index.html/#!/ssoLogin).

In TheHive 4.0 RC-2, OAuth 2 partially worked, and failed to redirect the user to the home page after the authentication success. Yes, sorry for that.

We spent some time testing the new implementation. We will devote some blog posts to it, but firstly, here is a working example relying on Keycloak

auth {
  providers: [
    {name: session}               # required !
    {name: basic, realm: thehive}
    {name: local}
    {name: key}    
    {
      ##############
      # Keycloak
      ##############
      name: oauth2
      clientId: "CLIENT_ID"
      clientSecret: "CLIENT_SECRET" # or empty
      redirectUri: "http://THEHIVE/api/ssoLogin"
      responseType: "code"
      grantType: "authorization_code"
      authorizationUrl: "http://KEYCLOAK/auth/realms/TENANT/protocol/openid-connect/auth"
      authorizationHeader: "Bearer"
      tokenUrl: "http://KEYCLOAK/auth/realms/TENANT/protocol/openid-connect/token"
      userUrl: "http://KEYCLOAK/auth/realms/TENANT/protocol/openid-connect/userinfo"
      scope: ["openid", "email"]
      userIdField: "email"
    }
  ]
}

After a question asked on Twitter, we tried to test our OAuth implementation with the providers mentioned in the answers, and we have successfully tested:

Migration tool performance

The migration tool we implemented in TheHive 4.0 RC-2 suffered from important performance issues as a result of our desire for a clean design.

In fact, enabling database locks during a parallelised and asynchronous processing of the migration operations produce a migration tool with poor performance.

We changed the strategy, by disabling locks and programmatically handling duplicates if they happen. This ended by a significant improvement of performance

We hope you can test it and provide us with your feedback.

Case sharing overview

Case sharing is the most important feature that the multi-tenancy support adds to TheHive. Allowing users to quickly spot if a case is owned or is coming from a share (made by another organisation) improves the user’s experience.

The other handy information is: the number of organisations having access to a certain case

Case list with sharing indicators

This screenshot shows all the case sharing related UI element:

  • The blue line, indicates that the case is coming from another organisation
  • The green line, indicates that the case is owned by the current organisation
  • The red line, highlights the column that show the number of organisation having access to the corresponding case

How to report issues

Please open an issue on GitHub using the template made for TheHive4 if you’d like to report a bug on this version. We will monitor those closely and respond accordingly.

TheHive 4.0-RC2, Fresh out of the Oven

Shortly after the release of TheHive 4.0 RC-1 in February 2020, many members of our community tested it and provided great feedback, spotting issues here and there. We would like to wholeheartedly thank all of those who, like us, want to make TheHive 4.0 a great, rock-solid release!

We are now happy to unveil the 2nd release candidate. It fixes many bugs and introduce – or reintroduce – some new (and old) features :-). In this blog post, we will concentrate on the following features:

  • 2FA
  • Age of cases
  • Reintroduction of webhooks

Please read the changelog for a more comprehensive view, including bugfixes.

And since the COVID-19 crisis is here to stay for quite some time, we don’t want you to rediscover boredom, a dreadful feeling long forgotten thanks to the continued stream of notifications, solicitations and attention-grabbing, 280 chars ‘thoughts’. So instead of getting bored, we invite you to test TheHive 4.0-RC2 to the best extent possible and, should you encounter any issue, please let us know. We want to issue the final release during the summer so that everyone can have it just in time for their forthcoming vacations at home!

2FA

Two factor authentication was initially scheduled for the final release. We changed our minds and decided to offer you the possibility to test this feature right away to gather your feedback and improvement ideas before we finish up baking the final recipe.

Users can enable 2FA from their account. To enable it, first go to your account Settings and check Enable Multi-Factor Authentication.

Once done, you are invited to use your preferred TOTP application (Google Authenticator, Authy, Microsoft Authenticator etc.) to scan the QR code or the code underneath it. Your 2FA will generate A TOTP that you should supply in the MFA Code area. If it is valid, 2FA will be activated.

Important notes:

  1. If a user loses access to their TOTP application, only an administrator can restore access to their account.
  2. If an org administrator loses access to their TOTP application and they are the only administrator for that org, only a super admin can restore access to their account.
  3. If a super admin loses access to their TOTP application and they are the only super admin of the instance, they should pack up their things and look for another job. That or use a magic DB command to restore access to their account. We’ll update the documentation accordingly.
  4. The current implementation of 2FA does not support backup codes or alternate authentication methods should a user loses access to their TOTP application. However, we are considering adding backup codes to the final release.
  5. 2FA cannot be enforced by default for all users at this stage. It is thus of rather marginal value. However, an org admin can see from the UI who did not activate it and pester them until they do. In the same way, a super admin can do the same for org admins, other super admins and mere users. We are updating the documentation to add an API query that will allow you to list all users who did not activate 2FA.
  6. We will consider making 2FA mandatory in TheHive 4.1.
2FA configuration view

Next time you log in, you will need to supply the TOTP verification code in addition to your login and password.

TOTP verification code required at login

Age of Cases

A new information regarding case duration has been added in the list of cases and in case view, so you can easily keep an eye on how old your cases are and activate your escalation procedures etc. if necessary.

Age of Cases in list view
Age of a Case in Case view

Webhooks are back!

TheHive 4.0-RC1 was released without webhooks. They have been reintroduced in this version. You can now configure TheHive 4.0 to use them, but also filter data sent to the remote server by Organisation.

How to report issues

Please open an issue on GitHub using the template made for TheHive4 if you’d like to report a bug on this version. We will monitor those closely and respond accordingly.

Under the Mighty Hood of TheHive 4

We have been speaking about it for almost two years. We have been making it for more than twelve months. And the day (or rather the month in this case) has almost come for TheHive 4, our latest and greatest version, to be unleashed.

While the first release candidate should be published by the end of this month, we would like to cover some of the most important changes we introduced in a platform which we rewrote almost from the ground up (40,000 lines of Scala code and counting), while keeping the familiar look&feel our longtime users came to expect.In a previous blog post, we covered TheHiveFS, a nifty feature of TheHive4 that allows you to quickly access all files stored in TheHive directly from your investigation machine. It’s time now to get a look under the hood of THeHive 4.

My Time is Precious. TL;DR Please

A picture is worth a thousand words, right? Here you go then!

The Hive 4’s Brand New Architecture

I am Puzzled, can you Elaborate a Bit?

So, you are not in a hurry anymore? Fine. Here, grab a seat, a glass of Gevrey-Chambertin and tasty Burgundy snails. All set? Let’s start then!

TheHive 4 will be the first version to use a graph database instead of Elasticsearch. Yes, you read that correctly. TheHive 4 won’t support Elasticsearch anymore but fear not fearless cyberdefender. Your friendly bees will not leave you hanging. If you are already using TheHive 3.4.x, we will provide a migration tool that will move your existing data to the new storage system (with no losses or bit flips hopefully).

We haven’t decided to ditch Elasticsearch on a whim or because Thomas (Franco, not Chopitea nor the General) dropped his leftist hipster attitude for a tight, tailor-made dictator uniform straight out of Spain. For all its greatness, ES has some annoying limitations which prevented us from adding, in an elegant, haiku-like way important features such as multi-tenancy, RBAC and large file management, while laying the ground for the future (stop being curious, the future has not been invented yet and when we do invent it, we’ll let you know).

Using JanusGraph, TheHive 4 structures information in graphs and stores them in an Apache Cassandra database. All the files that you attach to task logs or add as observables are stored in a Hadoop Distributed File System (HDFS).

Thanks to this brand new architecture, TheHive 4 is horizontally scalable. You can add as many TheHive, Cassandra and HDFS nodes to your Security Incident Response Platform cluster and sustain whatever load you might be facing without a sweat. Who said FOSS can’t be ‘enterprise grade’ (whatever that means in marketing lingo)?

Tour d’Horizon of the Main Features

TheHive 4, boosted by all the passion and skills of Zen Master Franco and MC Adouani, will support, in addition to TheHiveFS:

  • Multi-tenancy
  • RBAC
  • 2FA
  • Web configuration
  • API versioning

We will cover some of these features in greater detail in future instalments. In the meantime, let’s take a ride in a helicopter and view the wonderful landscape laying before us from above. After you Messieurs-Dames, we are French gentlemen and gallantry is of the essence (except when we use the public transportation in Paris, then savages we become).

Multi-Tenancy

As in Cortex, you will be able to create multiple organisations within a single instance of TheHive 4. In addition, an organisation can decide to share a case or parts of it (say a task, some observables, etc.) with other organisations. That way, a peer organisation or a constituent can contribute to the investigation at hand, provide essential information, etc.

RBAC

TheHive 4 supports a large set of user permissions. Some pertain to administrators, others to users and there are also permissions that apply to connectors. For example, users can manage tasks but not observables. They can have the power to share a case or part of it with sister organisations and execute Cortex analyzers but not responders.

You will be able to create roles for users, and, at the organisational level, what we call shares. RBAC deserves its own blog post and we’ll get to it pretty soon.

2FA

Do you really want us to describe this one? Before you answer yes, we’d like to remind you that you are in a helicopter. Just sayin’.

‘They asked me to explain 2FA. So I helped them out of the helicopter. It was flying way above ground.’
Source: Berserk, FNAC.com

Web Configuration

Tired of using vi, Emacs or your favourite CLI editor for making configuration changes to TheHive’s application.conf? Tired of restarting the service to take into account those modifications? Then you will certainly go dance kizomba with Nabil all night long when we tell you that you don’t need to use vi & service (or whatever the kids are using these days) anymore!

Thanks to the new architecture, all the configuration will be stored in the underlying database and you will be able to edit it using the WebUI. TheHive will automatically take the changes into account and you won’t need to restart it.

We can feel your love here. Merci !

API Versioning

TheHive 4 adds API versioning and it will maintain backward compatibility with TheHive 3.4.x without preventing us from adding new features. TheHive4py will not be updated right away for TheHive 4 but thanks to the backward API compatibility, all existing feeders and programs that use the current version of TheHive4py will still work out of the box.

That’s all folks! Stay tuned for further news and, in the meantime, don’t be blue cuz’ the bees gonna take care of you.

TheHiveFS

TheHive Project’s Code Chefs, sweating under their toques, are working hard to deliver TheHive 4 as soon as feasible. The current target release date for the 1st release candidate (4.0-RC1) is Friday Feb 28, 2020.

While TheHive 4 will be the first release to support graph databases, multi-tenancy and Role-Based Access Control (RBAC), it will also have a nifty feature that can simplify the incident response and digital forensics workflows of our fellow cyberdefenders: TheHiveFS.

What is TheHiveFS?

Starting from TheHive 4, TheHive can be ‘mounted’ as a remote, WebDAV filesystem. The filesystem can be securely mounted if SSL/TLS is enabled.

Thanks to TheHiveFS, you can quickly access all files stored in TheHive directly from your investigation machine. This can speed up the time needed to triage and analyse evidence. 

What Types of Files Can I Access through TheHiveFS?

You can access, in read-only mode, all files attached to task logs and all observables which datatype is file, as long as you are allowed to do so. Indeed, TheHive 4 comes with RBAC so if, for example, you are not allowed to view a case or some file observables in a case, you won’t be able to access them using TheHiveFS, the same way as if you are using the WebUI.

Screenshot showing an analyst accessing file observables and files associated to tasks of case #40 using TheHiveFS

How Can I Mount TheHiveFS?

Assuming you have a WebDAV client, such as davfs2, use the following command line:

$ sudo mount -t davfs -o noexec https://myhiveinstance:9001/fs /mnt/dav/

You can also point your graphical file manager to:

dav(s)://myhiveinstance:9001/fs

You will need to authenticate using your username and password as if you were connecting to TheHive’s WebUI.

Mom, I’ve Just Stepped on a Landmine

Beware folks. When you download a file observable using TheHive’s WebUI, it will conveniently create a password-protected ZIP archive before handing you the file. This way, we avoid accidental double clicks that may lead to the infection and compromise of your workstation, which might reflect bad on you or force you to offer breakfast the next morning to all your fellow teammates.

There is no such protection if you use TheHiveFS. Let us repeat this so it sinks: there is no such protection if you use TheHiveFS.

If you mount TheHive’s filesystem and open by accident or by a great deal of will, as a true, hardcore fan of Russian roulette, a file observable that is in fact malware courtesy of your favourite bear, kitten, panda or eagle, you can’t blame your friendly bees. But we will empathise (and our empathy level is directly correlated to the amount of pains au chocolat you send our way).

You’ve been warned.

That Sounds Awesome! When Can I Try It?

As written above, you will be able to try TheHiveFS as soon as TheHive 4.0-RC1 is released and that’s currently planned for the end of February 2020.

You can cry, beg, try to bribe us with VC money, make the line at 3:00 AM in front of TheHive Store (there ain’t no such store, we are not Apple), this will not make us work any faster. But you can always cheer us up, hug us or just thank us. This means a lot to us and to the free, open source software flame we carry deep within our souls.

One More Thing…

While we aren’t Apple, we can mimic Steve to share one more information that will make TheHiveFS even more interesting by Q3-Q4 2020. We plan to add support for large file management in TheHive 4.1, the next major version after 4.0 as would Captain Obvious say. Thanks to this feature, you will be able to upload memory and disk images to TheHive and if your Internet line breaks, the upload will resume automatically. 

That’s all folks!

TheHive 3.4.0 & Cortex 3.0.0 Released

For many months, we have been concentrating our efforts on TheHive 4, the next major version of your favourite Security Incident Response Platform, which we’ll finally provide RBAC (or multi-tenancy if you prefer), a feature that Cortex had for quite some time now.

Source : dilbert.com © Scott Adams

As you well know, both TheHive and Cortex rely on Elasticsearch (ES) for storage. The choice of ES made sense in the beginning of the project but as we added additional features and had new ideas to give you the best experience possible, we faced several ES quirks and shortcomings that proved challenging if not outright blocking for making our roadmap a reality, including RBAC implementation in TheHive, a far more complex endeavour than RBAC in Cortex. Transitioning from ES to graph databases was necessary and since we want our existing users to have a smooth migration path, TheHive 4 (the first release candidate should come out of the oven by the end of the year) will support both ES and graph databases.

But while we were focusing on that, we completely lost sight of the end of life of ES 5.6 so we wrote an apology to you, our dear users, back in May.

Shortly after, we released TheHive 3.4.0-RC1, to add support for ES 6 (with all the breaking changes it has introduced). We also did the same for Cortex with the release of Cortex 3.0.0-RC3. We also took that opportunity to clear out some AngularJS technodebt we had.

We then asked you to take them for a spin and report back any bugs you find given that both versions had to support ES 5.6 and ES 6 to allow for proper migration.

After a few rounds of release candidates, we are pleased to announce the immediate availability of TheHive 3.4.0 and Cortex 3.0.0 as stable releases.

Before upgrading your existing software to these new versions, please make sure to read the blog post we wrote back in June. We invite you to pay great attention to the regressions that we were forced to introduce because of ES 6.

You should also note that, in addition to ES 6 support, Cortex 3.0.0 supports fully dockerised analyzers and responders. We’ll elaborate on this in a future blog post soon.

Changelogs

If you are interested in some nitty-gritty details, we invite you to read the relevant changelogs since our last post on the subject:

Running Into Trouble?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help as usual!