Tag: TheHive4py

TheHive Reloaded: 4.1.0 is out

We are proud and excited to annonce the availability of TheHive v4.1.0. This release is the new beginning of TheHive’s future, because all the upcoming features and enhancements will be based on this version, without the brakes.

4.1.0 is out after a significant work done during the last 6 months, interrupted by multiple events like:

  • supporting ES 7 in TheHive 3 and Cortex
  • supporting ES 7.11 in TheHive 3 and Cortex
  • 5 patch releases of TheHive 4.0
  • recent OVH datacenter incident (but this one was quickly fixed)

Our community is waiting for it, as we announced it will fix the performance issues that many of you were facing. The reason is that TheHive 4.0.x doesn’t include a database indexation component resulting on poor data fetching performance.

We are glad about this release, it includes tons of new features and improvements, across all the application.

The first and maybe, the main one, is the database indexation (#1731) that will significantly improve performance of queries such as lists, filters and sorting. As a consequence, you should experience a smoother browsing and usage of the application.

Amongst all other news features, we introduced:

  • Support for MISP taxonomies, in addition to custom tags and tag colour support;
  • Support for Case TTPs specified by MITRE ATT&CK framework;
  • Better and improved Case merging and deleting capabilities;
  • A customisable date and time format
  • A refined Case Templates management UI
  • Enhanced Case list UI with bulk actions
  • Enhanced Task list UI with bulk actions
  • A platform status page to get an overview of the platform health
  • Support for migration from TheHive 3.5.1 to 4.1
  • Support for authentication on Webhook endpoints

For those of you who want to know the full details, here is the change log

A word about the documentation

As you might know, TheHive has a documentation repository that includes content for TheHive 3 and TheHive 4, but the structure needs to be redefined. For that reason, we introduced a new documentation website that aims to be the single starting point of all the docs we produce for TheHive Projects’ products.

In the upcoming days, we will mark the old docs repositories as ARCHIVED and we will maintain the new docs repository.

To visit the new documentation website, go to
https://docs.thehive-project.org

More about the new features

Database indexation

Until now, we knew TheHive suffered from a performance issue with the database.

Technically, TheHive uses basic index mechanism embedded in JanusGraph. This indexes are simple to use and manage but they contain limitations. They only support equality lookups and cannot be used for sort (for example, this is not possible to simply look for cases with severity > LOW) . However, within TheHive, almost all lists – lists of cases, lists of alerts, tasks …- are sorted. So, getting a list means a scan of all the elements of the list, which have a heavy performance impact, particularly if the list is long.

In order to solve this issue, TheHive 4.1.0 comes with a new index engine, and indexes have to be stored outside the database. As a consequence you need to define and setup dedicated storage for these indexes.

  • If TheHive is used in a cluster mode, all servers must connect to a common index engine, which can be in a cluster mode or not. In this case, a new component must be installed for index management. Elasticsearch should be used, only as an index engine. For clusters, Elasticsearch is not used like it was with TheHive 3.x when it stored data.
  • If TheHive is used on a standalone server, Elasticsearch can be used, but a file based index engine – Lucene – can be preferred. The latter solution has less infrastructure impact – no component to install – and only requires configuration that indicate where index data is stored in the filesystem.

For more details about database indexing configuration, please refer to the documentation

MISP taxonomies support

TheHive 4 handles tags in a completely different way than TheHive 3, even if the APIs are still considering tags as simple strings. For example when you create a Case from TheHiv4py, you specify tags as a simple array of strings.

Behind the scene, TheHive transforms the tags into objects, with a namespace, a predicate and an empty value. For example if your create a case with a free tag src=mailbox, TheHive 4 creates a new Tag object as following:

{
    "namespace": "_freetag",
    "predicate": "src=mailbox",
    "value": null,
    "colour": "DEFAULT_COLOUR_FROM_THE_CONFIG"
}

This is where we introduced the support of MISP Taxonomies, allowing admin users with `manageTaxonomy` permission, to import tags defined in machinetag.json format, from the “Administration > Taxonomies page”

Taxonomies import dialog

Note: importing the full MISP taxonomies library can take some time (1 minute)

Once imported, you need to enable the taxonomies you need for use cases. All the taxonomies are disables by default. Enabling a taxonomy, make its tags available to all the organisations on the platform, so analysts can use the to tag Cases, Alerts, Observables.

Tags selector

Taxonomy tags can also bee used from any tag filter in Case, Alert and Observable lists:

Tag filter

Custom tags support

Custom tags or free tags, is the way we call free text tags associated with TheHive objects. Internally, custom tags are included into organisation related spécial taxonomy called `freetags`.

Custom tags are not shared across organisations, so existing users defining sensitive data in tags won’t suffer any data leakage issue.

In general, custom tags in TheHive are not supposed to user data and information. It’s not a best practice to use email adresses as tags. Custom fields are the right place for this type of data.

This topic needs a dedicated blog post, to share the best practices of using tags.

So, how do an org admin manage the custom tags? Well, TheHive 4.1 comes with a new UI section, under the organisation management page, allowing:

  • Listing all the custom tags
  • Filtering and sorting
  • Updating custom tag values and colours
  • Deleting custom tags
  • Displaying custom tag usage (# of cases, # of alerts, # of observables and # of case templates)
Custom tags management

For users migrating from TheHive 3, all the existing tags are imported as custom tags

Tactics, Techniques & Procedures, with MITRE ATT&CK

Supporting MITRE ATT&CK framework is one of the feature that has been postponed many time from TheHive. It’s a no brainer feature that any blue team oriented product must include.

In TheHive 4.1, we allow:

  • Importing the official MITRE ATT&CK attack patterns collection as defined in https://github.com/mitre/cti. In this version support the entreprise catalog, but the future goal is to allow managing multiple catalogs in a misp-galaxy-like manner.
  • Defining TTPs associated with TheHive Cases

Attack Pattern management

From the administration page, any user with managePattern permission is able to have access to a page where patterns can be imported, filtered, viewed.

Import attack patterns
Attck Pattern management page

Case TTPs

In addition to Tasks and Observables, in TheHive 4.1, you can associated TTPs to your Cases. TTPs objects are defined by:

  • A tactic
  • A technique or sub-technique
  • An occur date
  • An optional procedure description
Add a TTP to a Case

The Case TTPs are displayed in a dedicated tab on the Case details page, the same way as Tasks and Observables, with filtering and sorting capabilities.

List of TTPs published in the report “SANDWORM INTRUSION SET
CAMPAIGN TARGETING
CENTREON SYSTEMS” by CERT-FR

This screenshot, showcases the tactic colours we use, thanks to Paul Tol’s blogpost

Case list TTP related improvements

One more feature: Case list now show the number of TTPs associated with each Case, with a link to the TTPs list page.

Case overview

Customise date and time

Since the very beginning, TheHive displays dates with the Month-Day-Year format. After 5 years, you can now customise the way you want date and time be displayed in all views of the UI.

This can be configured at the Organisation level, by defining the preferred format in the UI Configuration view. This required a user with org-admin profile or any profile with manageConfig permission.

UI Configuration view

Improved Case templates management

Case template management UI has been rewritten to allow more capabilities, required when you start having a growing number of templates:

  • Filtering
  • Sorting
  • Display dates
  • Display number of tasks and custom fields
Case template management UI
Case template editing UI

Case merging

This is a feature that has been removed from TheHive 4.0 as it required a design update to take into account the multi-tenancy support.

In TheHive 4.1, case merging has the same UI:

  • Go to a case details page
  • Click merge
  • Select the case to merge data into
  • Validate

The difference in TheHive 4.1 is that, merging two cases, removes the originating cases, and create a new one with all the merged data.

Platform status page

This feature aims to help understand the issues related to TheHive health status. It contains details about:

  • database schema version
  • status of database indexes
  • status of database integrity checks

It also allows:

  • exporting a JSON report of the health status including more details then what is displayed on the UI
  • reindexing the database
  • triggering the database integrity checks

This page is available to super admin users and contains cross organisations data

Platform status

Webhook authentication options

In TheHive 4.1, you can define authentication configuration for your Webhooks. For more details, please refer to the documentation website

KNOWN ISSUE: the auth property in Webhook definitions, in application.conf file is REQUIRED, so if your webhook doesn’t need authentication, then just add

auth: {type: "none"}

for each Webhook definition

Updating from TheHive 4.0

Due to the new database indexation feature, updating from 4.0.x requires some attention. Depending on your type of installation – standalone server or cluster – your setup will need to be upgraded accordingly.

  • On standalone servers, you will have to define a new local folder to store indexes;
  • For a cluster, you will need to locate an Elasticsearch instance to use it as index.

Finally, update the /etc/thehive/application.conf configuration file, install the new version of the application and restart the service.

more detailed information regarding this update can be found here:

https://docs.thehive-project.org/thehive/operations/update/

Indexes related to existing data will be created at the first start after the update to TheHive 4.1.0. Depending on the size of your database, this process can take a long time.

Upgrading from TheHive 3.x

If you are still using TheHive 3.x and want to migrate to TheHive 4.1, then you need to take a look to the following supported paths:

Target versionRequired source version
TheHive 4.1.xTheHive 3.5.1
TheHive 4.0.x3.4.x < TheHive < 3.5.1
Supported migration paths

For more details, please refer to the documentation website

How to install

If you starts using TheHive with this version, we recommend having a look at our documentation site, and particularly to the installation and configuration section which is up to date and contains all instructions to install & configure TheHive 4.1.0.

Docker

If you use TheHive as a docker container, you can refer to the Docker-Templates repository that has a TheHive 4 up-to-date docker-compose configurations.

How to report issues

Please open an issue on GitHub with the dedicated template for TheHive 4. We will monitor them closely and respond accordingly. 

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Discord, or send us an email at support@thehive-project.org. We will be more than happy to help!

TheHive 4.0.4 and TheHive4py 1.8.1: alerts got more APIs

Dear community, the new year has brought us another opportunity to build new features in your favorite Security Incident Response Platform, TheHive. We wish you a cheerful new year ahead and we thank you for being beside us all these years.

Last week, we released TheHive 4.0.4 and TheHive4py 1.8.1, and here is the official announcement including the details of the new features.

These releases focused on adding more capabilities to play with alert observables and give more flexibility when building alert feeders.

Please find the change logs for more details:

What’s new in TheHive

New Alert observable APIs

The major change in TheHive 4.0.4 is related to alert management. In TheHive 3, alert observables were included in the alert as an array of observable objects, and not as independent objects with links to the alert itself. This data model made alert observables CRUD operations, a bit challenging.

TheHive 4 has a better design for this, and alert observables have their own existence, and can be added/updated and deleted independently from the alert object.

This new design allows adding dedicated API endpoints to:

  • Add an observable to an existing alert;
  • Update the data of an existing alert observable;
  • Delete an observable from an alert.

Those APIs are not used by the user interface for now.

New Alert properties

This release introduced a new property called `importDate`. It represents the date at which an alert has been merged into a new/existing case. This property is then used to:

  • Allow filtering the alert list, for example: “List the alerts merged today”
  • Display the duration between the alert creation and its merge into a case.
Alert list showcasing alert importDate

This new property is of course available on the dashboard creation UI as a date field, among others:

  • imported: true if the alert has been merged
  • `handlingDurationInSeconds`: number of seconds before importing an alert
  • `handlingDurationInMinutes`: number of minutes before importing an alert
  • `handlingDurationInHours`: number of hours before importing an alert
  • `handlingDurationInDays`: number of days before importing an alert

To showcase the mentioned new properties, here are some screenshots:

importDate field used on a line chart as date field for x-axis
Simple line chart using the imported filter
Dashboard including two charts using the newly introduced alert fields

What’s new in TheHive4py

The 1.8.1 release of TheHive4py mainly focuses on adding support to the new alert APIs introduced by TheHive 4.0.4. It comes with 3 new functions:

  • `create_alert_artifact` to allow developers adding a new artifact to an existing alert
from thehive4py.api import TheHiveApi
from thehive4py.models import Tlp

THEHIVE_URL = 'http://127.0.0.1:9000'
THEHIVE_API_KEY = '**YOUR_API_KEY**'

api = TheHiveApi(THEHIVE_URL, THEHIVE_API_KEY)

# Instanciate a new domain artifact
artifact = AlertArtifact(dataType='domain', data='malicious-domain.tld', ignoreSimilarity=True, ioc=True)
api.create_alert_artifact(ALERT_ID, artifact)

# Instanciate a new file artifact
artifact = AlertArtifact(
    dataType='file', 
    data='malicious-file.exe', 
    ignoreSimilarity=False, 
    ioc=True, 
    sighted=True, 
    tlp=Tlp.RED.value)
api.create_alert_artifact(alert_id, artifact)
  • `update-alert-artifact` to allow updating the data of an existing alert artifact:
from thehive4py.api import TheHiveApi
from thehive4py.models import Tlp

THEHIVE_URL = 'http://127.0.0.1:9000'
THEHIVE_API_KEY = '**YOUR_API_KEY**'

api = TheHiveApi(THEHIVE_URL, THEHIVE_API_KEY)

# Create a new domain artifact
artifact = AlertArtifact(dataType='domain', data='malicious-domain.tld', ignoreSimilarity=True, ioc=True)
response = api.create_alert_artifact(ALERT_ID, artifact)

# Update its tlp, sighted and ignoreSimilarity flags
artifact_data = response.json()[0]
artifact_data['tlp'] = Tlp.RED.value
artifact_data['sighted'] = True
artifact_data['ignoreSimilarity'] = False

new_artifact = AlertArtifact(json=artifact_data)
api.update_alert_artifact(artifact_data['id'], new_artifact, fields=['tlp', 'ioc', 'ignoreSimilarity'])
  • `delete_alert_artifact` to allow removing an existing artifact from an existing alert
from thehive4py.api import TheHiveApi

THEHIVE_URL = 'http://127.0.0.1:9000'
THEHIVE_API_KEY = '**YOUR_API_KEY**'

api = TheHiveApi(THEHIVE_URL, THEHIVE_API_KEY)

# Delete alert artifact
api.delete_alert_artifact(ARTIFACT_ID)

Note that these new three methods are only available when using TheHive4py with TheHive 4.0.4+

You can find more details on the official documentation of TheHive4py.

Updating/Installing

To update your existing package to version 1.8.1:

$ sudo pip install thehive4py --upgrade

How to report issues

Please open an issue on GitHub with the dedicated template for TheHive 4 or the dedicated form for TheHive4py. We will monitor them closely and respond accordingly. 

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Discord, or send us an email at support@thehive-project.org. We will be more than happy to help!

TheHive4py 1.8.0 is hot off the press

TheHive4py 1.8.0 is finally released. During the last 5 months, the team was busy working on TheHive and Cortex but today, it’s time to unveil the biggest milestone of TheHive’s official python API client.

TheHive4py official documentation website

1.8.0 contains 31 Github issues including 17 contributions for which, we would like to thank all the community members who helped shaping the release.

TheHive4py is getting a bit hard to maintain because of the backward compatibility constraints introduced by TheHive 3 and 4 versions. TheHive 4 also introduces new features that are not available in TheHive 3, and this makes it challenging to serve both versions with the same code base.

TheHive 4 has also its dedicated and optimised APIs (read APIs v1), and those are not used by TheHive4py, which is still relying on APIs v0 of TheHive 4.

Anyways.

What’s new?

1.8.0 release introduced a significant number of new methods and changes:

  • attachment download support for files (by id), observables and task log attachments
  • alert merge into case
  • alert delete
  • case task delete
  • case task log search
  • task log search
  • support to alert similarity in fetch
  • case observable search method
  • case observable fetch method
  • case observable delete method
  • support to in memory files when calling APIs evolving attachments
  • MISP export
  • support to PAP in alerts
  • add Tlp, Pap, Severity, CaseStatus, TaskStatus enumerations

in addition to some TheHive 4 related features:

  • Add a version parameter to TheHiveApi class’s constructor
  • Add support to ignoreSimilarity attribute
  • Add support to alert.externalLink attribute

Please read the full release notes for more details.

Below, we will highlight the major features other than the self explanatory newly added methods.

New version parameter

This change is important and required for developers using TheHive4py to play with a TheHive 4 instance. the `version` parameter has been introduced to allow fine tune access to features available on TheHive 4 and not in TheHive 3, like for `alert.extrnalLink` field.

The version is set by default to Version.THEHIVE_3.value, which means version 3.

from thehive4py.models import Version

# Init an API client for TheHive 4
api = TheHiveApi(THEHIVE_URL, API_KEY, version=Version.THEHIVE_4.value)

Add support to ignoreSimilarity field

This capability has been introduced by TheHive 4.0.1 release. It allows setting an `ignoreSimilarity` flag at the case and alert observable level. When set to True it tells TheHive to ignore the observable from any similarity computing.

So, if you need to create an alert with an observable you would like to skip when running the similarity algorithm, then, you need to set ignoreSimilarity to True

Here is an example that creates and alert with an observable to be ignored for similarity:

import uuid
from thehive4py.api import TheHiveApi
from thehive4py.models import Tlp, Pap, Alert, AlertArtifact

sourceRef = str(uuid.uuid4())[0:6]

# Prepare the Alert object
alert = Alert(title='Sample alert - ID {}'.format(sourceRef), 
  tlp=Tlp.AMBER.value, 
  pap=Pap.AMBER.value, 
  tags=['TheHive4Py'],
  description='Sample alert for the blog post',
  source='dev',
  type='script',
  sourceRef=sourceRef,
  externaleLink='https://some-web-site/alert/{}'.format(sourceRef),
  artifacts=[
    AlertArtifact(
      dataType='domain', 
      data='dl.some-web-site.com', 
      tlp=Tlp.WHITE.value, 
      ioc=True,
      sighted=False,
      ignoreSimilarity=True
    )
  ])

# Init an API client instance
api = TheHiveApi(THEHIVE_URL, THEHIVE_API_KEY, version=4)

# Create the alert
response = api.create_alert(alert)

Note that the same option is available for Case observables during creation and update:

# Set an observable as ignorable

api.update_case_observable(
   observable_id, 
   {"ignoreSimilarity": True}, 
   fields=['ignoreSimilarity']
 )

Support in-memory files

Old versions of TheHiv4py required existing files when dealing with attachments. For example, to create a file case observable, the corresponding file has to be already stored on the file system before calling the `create_case_observable` method.

This release allows using files from memory and not relying on file paths. So you use, for example, an API to download a file and you want to store that file as observable, you can use the new feature of TheHive4py, as below:

from thehive4py.models import Tlp, Pap, Alert, CaseObservable

# Say you have a method to get a screenshot
file = get_screenshot()

# Prepare the observable
observable = CaseObservable(
    dataType='file',
    data=(file, 'screenshot-{}.png'.format(int(time.time())*1000)),
    tlp=Tlp.WHITE.value,
    pap=Pap.GREEN.value,
    tags=['category:screenshot']
)

# Create the observable
response = api.create_case_observable(case_id, observable)

# Close the file object
file.close()

Note: closing the file object is still required. We will handle closing the files during the upcoming releases.

Attachment download features

The new methods introduced by 1.8.0 release and related to attachment download, light some interesting TheHive APIs up.

Did you know TheHive has APIs to download existing file from the datastore? Do you know how does TheHive store files?

Well, when a file is uploaded to TheHive as case observable, alert observable or case log attachments, the file is stored in the DataStore and is given an ID that can be used in two APIs:

  • `/api/datastore/{attachment_id}`: downloads the file content
  • `/api/datastorezip/{attachment_id}`: downloads the file content as zip password protected file, and the password is the one defined in application.conf (defaults to `malware`)

In this release, we introduced methods to make these APIs available on TheHive4py.

Download an attachment by its ID

This method is useful if you know the attachment ID (could be an alert file observable for exemple, in TheHive 4)

# Download an attachment by a known id
response = api.download_attachment(
  attachment_id, 
  filename='screenshot.png',
  archive=False
)

# Save the attachment to disk
f = open('./{}'.format('screenshot.png'), 'wb')
f.write(response.content)
f.close()

Download an attachment of a task log

This method allows downloading the attachment of a given task log object, identified by its ID

response = api.download_task_log_attachment(log_id, archive=False)

f = open('./{}'.format('screenshot.png'), 'wb')
f.write(response.content)
f.close()

If the task log doesn’t have an attachment, this methods throws an exception.

Download a file attachment of a file observable

This methods allows downloading a file observable, and forces protecting it as password protected zip archive

response = api.download_observable_attachment(observable_id)

f = open('./{}'.format('observable.png.zip'), 'wb')
f.write(response.content)
f.close()

If the observable is not a file, this methods throws an exception.

Documentation

As you might know, with the 1.7.0 release we released a documentation website for TheHive4py where all the methods are documented. Donc hesitate to refer to it for more details: https://thehive-project.github.io/TheHive4py

Updating/Installing

To update your existing package to version 1.8.0:

$ sudo pip install thehive4py --upgrade

Got a question?

If you encounter any difficulty, please join our  user forum, contact us on Discord, or send us an email at support@thehive-project.org. As usual, we’ll be more than happy to help!

TheHive4py got a second wind, version 1.7.0 released

“TheHive4py”, this sounds like a word you didn’t hear about during the last 12 months. Well, our focus on this library was put on hold. We will tell you the reason, but much better, we will solve the problem.

A brief review

TheHive4py was quickly initiated after the first releases of TheHive to help developers interact with TheHive APIs using python. We started creating methods and functions for main functionalities and to be honest, it was a sort of a quick-and-dirty solution.

TheHive4py has some limitation:

  • The API client is a flat class with dozens of methods
  • The API clients’ methods return the native `requests.Reponse` class instead of a structured data
  • Exception handling could be improved
  • Code could be made more reusable

As developers, we are aware of these limitations and are eager to provide a better library, and that’s what we started making with TheHive4py rewrite. We wanted to provide you with a library you can use this way:

# Fetch cases
open_cases = api.cases.find_all({'status': 'Open'}, range='0-5')
log('Open cases', list(map(lambda i: i.json(), open_cases)))

# Fetch a case by `id` or `number` (caseId)
sample_case = open_cases[0]
log('case details by id', api.cases.get_by_id(sample_case.id).json())
log('case details by number', api.cases.get_by_number(sample_case.caseId).json())

# Fetch alerts
new_alerts = api.alerts.find_all({'status': 'New'}, range='0-2')
log('New alerts', list(map(lambda i: i.json(), new_alerts)))

# Fetch observables
domain_observables = api.observables.find_all({'dataType': 'domain'}, range='0-2')
log('New alerts', list(map(lambda i: i.json(), domain_observables)))

# Fetch tasks
waiting_tasks = api.tasks.find_all({'status': 'Waiting'}, range='0-2')
log('Waiting tasks', list(map(lambda i: i.json(), waiting_tasks)))

waiting_tasks = api.tasks.get_waiting(range='0-2')
log('Waiting tasks', list(map(lambda i: i.json(), waiting_tasks)))

jdoe_tasks = api.tasks.get_by_user('jdoe', {}, range='0-3')
log('Tasks of jdoe', list(map(lambda i: i.json(), jdoe_tasks)))

case_tasks = api.tasks.of_case(sample_case.id, query={'status': 'Waiting'})
log('Case tasks', list(map(lambda i: i.json(), case_tasks)))

The library’s rewrite was supposed to produce a 2.0.0 version of TheHive4py but we had a major issue: backward compatibility.

Well, in theory, backward compatibility can be handled through a clear communication to:

  • tell the users how to make sure to update their dependencies to TheHive4py < 2.0.0
  • provide a migration plan
  • maintain both versions during a certain time
  • maintain documentation for old and new versions

To be honest, this was hard to achieve, because of the famous lack of time, but things a going to change.

What’s the plan?

We didn’t want to make a plan without asking the community about how they interact with TheHive APIs. So we did two twitter polls that ended up with the following results:

Twitter poll about TheHive API usage methods

The second poll asked our users about pros and cons of TheHive4py:

Twitter poll about TheHive4py pros and cons

The poll results are clear: we need to put more efforts on TheHive4py.

Here we go, firstly, let’s release version 1.7.0

TheHive4py 1.7.0 milestone has been initiated almost one year ago, and we are happy to announce its availability today.

What’s new about it?

The most important change is allowing TheHive4py to interact with TheHive 4 in addition to introducing some missing features, and bug fixes. Here is a short listing of main changes:

Add support to multi tenancy

Allow a developer to specify the organisation against which an API call is done:

api = TheHiveApi('http://my_thehive:9000', 'my_api_key', organisation='cert')

Add custom field support for new types:

TheHive 4 introduces custom fields of type integer and float, this feature allows specifying custom fields with types supported by TheHive 4. These types are not supported by TheHive 3.

CustomFieldHelper
   .add_integer('number_hits', 10)
   .add_float('cvss', 5.6)
   .build()

The code snippet above produces the following content:

{
  "number_hits": {
    "order": 0,
    "integer": 100
  },
  "cvss": {
    "order": 1,
    "integer": 5.6
  }
}

Add support to like and wildcard query operators

TheHive query DSL supports like and wildcard operators, but TheHive4py didn’t had an option to use those operators. In this version the following query methods have been added:

  • Like (field, value): Field’s value must contain value, that must contain `*` in the beginning or at the end
  • StratsWith (field, value): Field’s value must start with value
  • EndsWith (field, value): Field’s value must end with value
  • ContainsString (field, value): Field’s value must contain value
from thehive4py.query import Eq, Like, And, StartsWith

# find cases where title contains 'Dridex'
api.find_cases(query=Like('title', 'Dridex*'))

# find alerts where status is 'New' and title starts with 'Emotet'
api.find_alert(query=And(Eq('status', 'New'), StartsWith('title', 'Emotet')))

Add ioc and sighted attributes to case and alert artifacts

This allows specifying these attributes during Alert or Case observables creation

Add update_case_observable method

Can be used to patch an existing observable, by setting a tag or marking as IOC.

Add PAP to Case and CaseTemplate models

PAP flag has been added in TheHive recently and TheHive4py was not able to set the PAP value of a Case or CaseTemplate

Add custom fields creation method

Added a `create_custom_field` method that check custom field name uniqueness before creating it.

Note: This method is for now, compatible with TheHive 3 only because it relies on the DBList API that is no longer available on TheHive 4.

Add case template creation method

Added a `create_case_template` method allowing developers to create new Case Templates.

The full change log is available at the release page

What about documentation

Once again we are glad to announce the initial version of a documentation website, dedicated to TheHive4py, including documentation of all the features the library provides, and code samples of the most useful features.

We aim to maintain and improve this documentation over time, so please, don’t hesitate to either contribute or ask for more content.

Screenshot of the documentation website

TheHive4py 2.0

We will put the rewrite of TheHive4py on hold for now and will communicate about it again when we are ready. In the meantime, we will continue maintaining TheHive4py 1.x.

Update: TheHive4py 1.7.1 Patch

During the release 1.7.0, we have noticed that the build process and deployment went wrong, so we have created a 1.7.0.post1 release.

The community also raised a regression that has been fixed in 1.7.1 release. You can read the change log for more details.

Updating/Installing

To update your existing package to version 1.7.0:

$ sudo pip install thehive4py --upgrade

Got a question?

If you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. As usual, we’ll be more than happy to help!

Under the Mighty Hood of TheHive 4

We have been speaking about it for almost two years. We have been making it for more than twelve months. And the day (or rather the month in this case) has almost come for TheHive 4, our latest and greatest version, to be unleashed.

While the first release candidate should be published by the end of this month, we would like to cover some of the most important changes we introduced in a platform which we rewrote almost from the ground up (40,000 lines of Scala code and counting), while keeping the familiar look&feel our longtime users came to expect.In a previous blog post, we covered TheHiveFS, a nifty feature of TheHive4 that allows you to quickly access all files stored in TheHive directly from your investigation machine. It’s time now to get a look under the hood of THeHive 4.

My Time is Precious. TL;DR Please

A picture is worth a thousand words, right? Here you go then!

The Hive 4’s Brand New Architecture

I am Puzzled, can you Elaborate a Bit?

So, you are not in a hurry anymore? Fine. Here, grab a seat, a glass of Gevrey-Chambertin and tasty Burgundy snails. All set? Let’s start then!

TheHive 4 will be the first version to use a graph database instead of Elasticsearch. Yes, you read that correctly. TheHive 4 won’t support Elasticsearch anymore but fear not fearless cyberdefender. Your friendly bees will not leave you hanging. If you are already using TheHive 3.4.x, we will provide a migration tool that will move your existing data to the new storage system (with no losses or bit flips hopefully).

We haven’t decided to ditch Elasticsearch on a whim or because Thomas (Franco, not Chopitea nor the General) dropped his leftist hipster attitude for a tight, tailor-made dictator uniform straight out of Spain. For all its greatness, ES has some annoying limitations which prevented us from adding, in an elegant, haiku-like way important features such as multi-tenancy, RBAC and large file management, while laying the ground for the future (stop being curious, the future has not been invented yet and when we do invent it, we’ll let you know).

Using JanusGraph, TheHive 4 structures information in graphs and stores them in an Apache Cassandra database. All the files that you attach to task logs or add as observables are stored in a Hadoop Distributed File System (HDFS).

Thanks to this brand new architecture, TheHive 4 is horizontally scalable. You can add as many TheHive, Cassandra and HDFS nodes to your Security Incident Response Platform cluster and sustain whatever load you might be facing without a sweat. Who said FOSS can’t be ‘enterprise grade’ (whatever that means in marketing lingo)?

A Tour d’Horizon of the Main Features

TheHive 4, boosted by all the passion and skills of Zen Master Franco and MC Adouani, will support, in addition to TheHiveFS:

  • Multi-tenancy
  • RBAC
  • 2FA
  • Web configuration
  • API versioning

We will cover some of these features in greater detail in future instalments. In the meantime, let’s take a ride in a helicopter and view the wonderful landscape laying before us from above. After you Messieurs-Dames, we are French gentlemen and gallantry is of the essence (except when we use the public transportation in Paris, then savages we become).

Multi-Tenancy

As in Cortex, you will be able to create multiple organisations within a single instance of TheHive 4. In addition, an organisation can decide to share a case or parts of it (say a task, some observables, etc.) with other organisations. That way, a peer organisation or a constituent can contribute to the investigation at hand, provide essential information, etc.

RBAC

TheHive 4 supports a large set of user permissions. Some pertain to administrators, others to users and there are also permissions that apply to connectors. For example, users can manage tasks but not observables. They can have the power to share a case or part of it with sister organisations and execute Cortex analyzers but not responders.

You will be able to create roles for users, and, at the organisational level, what we call shares. RBAC deserves its own blog post and we’ll get to it pretty soon.

2FA

Do you really want us to describe this one? Before you answer yes, we’d like to remind you that you are in a helicopter. Just sayin’.

‘They asked me to explain 2FA. So I helped them out of the helicopter. It was flying way above ground.’
Source: Berserk, FNAC.com

Web Configuration

Tired of using vi, Emacs or your favourite CLI editor for making configuration changes to TheHive’s application.conf? Tired of restarting the service to take into account those modifications? Then you will certainly go dance kizomba with Nabil all night long when we tell you that you don’t need to use vi & service (or whatever the kids are using these days) anymore!

Thanks to the new architecture, all the configuration will be stored in the underlying database and you will be able to edit it using the WebUI. TheHive will automatically take the changes into account and you won’t need to restart it.

We can feel your love here. Merci !

API Versioning

TheHive 4 adds API versioning and it will maintain backward compatibility with TheHive 3.4.x without preventing us from adding new features. TheHive4py will not be updated right away for TheHive 4 but thanks to the backward API compatibility, all existing feeders and programs that use the current version of TheHive4py will still work out of the box.

That’s all folks! Stay tuned for further news and, in the meantime, don’t be blue cuz’ the bees gonna take care of you.

DigitalShadows Feeder 2.4.5

DigitalShadows2TH 2.4.5, a new version of the DigitalShadows feeder for TheHive, has been released with major improvements.

Source : https://www.bustle.com

Dockerise All The Things!

The most notable one is that you can now use it with docker. Run docker pull thehiveproject/ds2th and create a homedir for configuration and logs.

$ DS2TH_HOMEDIR = /opt/thehive_feeders/Digitalshadows2TH/
$ docker pull thehiveproject/ds2th:latest
$ mkdir -p $DS2TH_HOMEDIR/{config,log}
$ wget -O $DS2TH_HOMEDIR/config/config.py\
https://raw.githubusercontent.com/TheHive-Project/DigitalShadows2TH/master/config.py.template

Edit the config.py before runing the docker command below: 

$ docker run --rm --net=host --mount\
type=bind,source="$DS2TH_HOMEDIR"/config,target=/app/config --mount type=bind,source="$DS2TH_HOMEDIR"/log,target=/app/log certbdf/ds2th <OPTIONS>
Quick Installation

If that sounds still complicated, worry not! We also wrote a little program that helps you prepare the environment and install everything. We ensured that it works well on Ubuntu 18.04. The program uses two environment variables to set up everything: FEEDERS_SYSACCOUNT and FEEDERS_HOMEDIR : 

$ export FEEDERS_SYSACCOUNT=thehive
$ export FEEDERS_HOMEDIR=/opt/thehive_feeders

There are also sane, default settings in case you did not set any value. DigitalShadows2TH’s home directory will be set to /opt/thehive_feeders/DigitalShadows2TH. To use the script, run the following command line and follow the instructions:

 $ wget -qO- https://raw.githubusercontent.com/TheHive-Project/DigitalShadows2TH/mater/INSTALL/install_with_docker.sh | sudo -E bash -

Custom Case Templates

Previous versions of DigitalShadows2TH allowed only one case template to be associated with alerts created by the feeder in TheHive. Starting from DigitalShadows2TH 2.4.0, you can define a case template for each type of incidents raised by DigitalShadows in the configuration file.

The configuration pertaining to TheHive looks as follows:

TheHive = {
  'proxies':{
      'http': '',
      'https': ''
  },
    'url':'THEHIVE_URL',
    'key':'THEHIVE_API_KEY',
    'templates': {
        'default':''
    }
}

Each incident type in DigitalShadows can be associated with a case template in TheHive, for example:

TheHive = {
  'proxies':{
      'http': '',
      'https': ''
  },
    'url':'THEHIVE_URL',
    'key':'THEHIVE_API_KEY',
    'templates': {
        'default':'MY_DEFAULT_CASE_TEMPLATE_FOR_DIGITALSHADOWS_INCIDENTS',
        'DATA_LEAKAGE': 'MY_DATA_LEAKAGE_CASE_TEMPLATE',
        'CYBER_THREAT': 'MY_CYBER_THREAT_CASE_TEMPLATE'
    }
}

A template can be defined for all the following DigitalShadows incident types:

  • DATA_LEAKAGE
  • CYBER_THREAT
  • PHYSICAL_SECURITY
  • SOCIAL_MEDIA_COMPLIANCE
  • BRAND_PROTECTION
  • INFRASTRUCTURE

A default template can be defined for DigitalShadows incidents. If no template is found for a specific incident type, the feeder looks for the default template. if no default template is found, an empty case will be created by when importing the alert.

Update or Install

If you are not using docker, just pull the repository and update your configuration file with the new templates part for TheHive.

Update your Repository
$ cd /opt/TheHive_feeders/DigitalShadows2TH/
$ git pull

The configuration file has changed, so you need to update yours before running the program. A new templates section has been added for TheHive and the path has changed. It is now in the config/ directory of the project.

Install and Use via the Code Repository
$ cd /opt/TheHive_feeders 
$ git clone https://github.com/TheHive-Project/DigitalShadows2TH.git

After that, follow the prerequisites and edit the configuration file. In /opt/TheHive_feeders/DigitalShadows2TH/config/ copy config.py.template to config.py and modify it.

Use cases and detailed configuration instructions can be found in the README file in the repository.

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

TheHive4py 1.5.1 Released

When you need to interact with TheHive’s REST API and you ain’t shy of working with Python, TheHive4py is the way to go. It’s a free, open source library we provide to allow you to easily create alert feeders, automate certain tasks like creating cases, assign them to analysts and much more. For example, Synapse, DigitalShadows2TH and Zerofox2TH leverage the library to send alerts to your favourite SIRP/SOAR.

Sometime ago, we decided that it was time to overhaul the whole library and we began working on version 2.0.0 which will be easier to use. It should also support the full set of TheHive’s REST API calls. In the meantime we decided to release version 1.5.0, shortly followed by version 1.5.1 to support some new functionality contributed by our user community and correct a few issues.

code_quality
Source : XKCD

New Features Introduced in 1.5.0

Bugfixes Introduced in 1.5.0

  • #80: Prevent max recursion depth exceeded error, contributed by Psynbiotik

New Features Introduced in 1.5.1

Important note: TheHive4py 1.5.1 does not work with TheHive 3.0.10 or earlier versions. Please stick with 1.5.0 if you are using those versions.

Updating/Installing

To update your existing package:

$ sudo pip install thehive4py --upgrade

If you are just getting started with TheHive4py, you can forgo the --upgrade at the end of the command above.

But I just Wanna Play!

If you’d like to play around with TheHive4py 1.5.1, TheHive 3.1.1., Cortex4py 2.0.1 and Cortex 2.1.1, please download the training VM.

Paris? Are you There?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. As usual, we’ll be more than happy to help!

Correction: October 12, 2018
As reported by Robin Hahling, TheHive 1.5.1 does not work with TheHive 3.0.10 or earlier versions.

Unveiling Synapse

When we’re not busy cooking new features, we go back to the trenches and face incidents like many of our fellow analysts who read our publications and use our tools. To do so, we swap our chef toques for firefighter helmets, not only because such shiny headwear is cool, but mainly because incident response (IR) is, at its very heart, firefighting (minus all the dangerous stuff).

If you think about it, when handling incidents you can see everything from cats in trees (spam) to major fire (APT). Thankfully, there are more cats to bring down than fire to extinguish. That being said, a big herd of cats could be a serious threat to your organization, to your mental health or both.

We tend to forget that incident handlers are humans, not robots. Unlike our metal cyberfriends, we need diversity. We can’t risk insanity like Charlie Chaplin in Modern Times if we can avoid it. Unfortunately IR can be highly repetitive, especially if you only have cats to deal with.

Some could say ‘Nah, this is minor, nothing critical here’ but at some point, an analyst brainwashed by the same tasks again and again will be led to fault. In the worst case scenario, one could see an alert and immediately categorize it as false positive without any further consideration. Because ‘this alert is always a false positive’, until the day it is not…

Automation, a Solution?

Intuitively, we look in the direction of automation in order to minimize what we call ‘zombie’ tasks: highly repetitive and brainless tasks that need to be done. We believe that doing so will allow incident handlers to focus on the analysis and not on the tedious side of IR. Ultimately, we hope it will keep analysts stimulated and in a state of alert. Also, it should reduce time and effort spent on the low-hanging fruits.

One of the most dreary tasks in our opinion is to record the context around an incident.
What is the problem? When did it happen? What’s the origin? Who are the victims? How many are there? Answers to these questions let you have an overview of what is happening and are valuable to correlate incidents. So it is worth taking some minutes to add this information to your case. Sadly, most of the time it will look like a succession of ‘Ctrl+C; Alt+Tab; Ctrl+V’ from your incident source to TheHive. Exactly the kind of tasks we want to forego.

Specifications

Having identified the threat that apathetic analysts pose, the root cause (highly repetitive tasks) and a solution (automate the recording of incident context), the question of the implementation has been raised.

The first challenge to solve is the number of incident sources. Almost everything can trigger an incident: a firewall, an IDS, antivirus, SIEM, users, etc… So the application must be designed to accept several sources and must permit to easily integrate new ones. And instead of having to configure multiple alert feeders to supply alerts to TheHive, we would have only one. To some extent, it can be assimilated to a meta feeder.

And if the application works as intended, we still have a second challenge. Let’s say you, dear reader, and ourselves use the galaxy renowned Stargazer IDS. Maybe you’d like to include the full packet capture in the case but we wouldn’t. Using the same product doesn’t mean using it the same way. So we have a variety of sources and for each source, we have a variety of configurations and workflows. Hence any app we design needs to accept multiple configurations and workflows for any given source.

Finally: the third challenge. We want to make the most out of TheHive. Creating cases, creating alerts, assigning cases, adding logs, adding observables … all those actions are not an option.

Synapse

After several trials and failures, we came up with Synapse. Basically it is a Python 3 app which sits between TheHive and your incident sources:

Screen Shot 2018-07-18 at 09.54.28.png
Synapse Overview

To solve the first and third challenge, we rely on connectors. A connector is a Python object dedicated to interact with a security device. In the picture above, you can see the Exchange Connector and TheHive Connector. To extend the number of sources, you just have to develop the connector that corresponds to your device.

Regarding the second challenge, we rely on workflows. Workflows are python scripts who use connectors to automate repetitive tasks when tracking a case. Not happy with the current workflow? Develop your own using the connectors.

At this point, you probably wonder why there’s an API in the picture above. Well, the API is the link between the user and the workflows. By hitting a specific endpoint of the Synapse’s API, the corresponding workflow will be launched. That way the user can choose what to launch, especially if they are only interested in a particular workflow. Moreover, using an API allows us to listen to TheHive’s real-time stream and initiate some actions like closing a QRadar offense when the related case is solved.

At the moment, Synapse includes the Exchange connector and the associated Ews2Case workflow. The workflow features:

  • Case creation from emails
  • Case assignment
  • Adding email bodies to task logs
  • Adding email replies to the case
  • Adding email attachments as observables

And of course, everything is done to minimize the number of clicks! Check the workflow documentation to understand how it works under the hood.

We’re still working on the QRadar connector and the associated workflows but if you can’t wait, have a look at the work done by the community like pierrebarlet’s script.

Check it Out

As usual, Synapse is an open source and free software released under the AGPL (Affero General Public License).

Synapse has its own repository. Start with the user guide and read about the workflow you want to use as you’ll need to configure it.

Troubles?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help!

TheHive4py 1.4.0 Released

Version 1.4.0 of the Python API client for TheHive is now available. It is compatible with the freshly released Cerana (TheHive 3.0.0).

We’d like to thank Nick Pratley, a frequent contributor, Bill Murrin, Alexander Gödeke and “srilumpa” for their code additions and documentation.

To update your existing package:

$ sudo pip install thehive4py --upgrade

If you are just getting started with TheHive4py, you can forgo the --upgrade at the end of the command above.

New Features

  • #5: Add a method to update a case, contributed by Nick Pratley
  • #34: Add a get_task_logs method in order to obtain all the task logs associated with a given taskId. Contributed by Bill Murrin
  • #37: A new, very cool case helper class by Nick Pratley
  • #39: Add support for custom fields to the case model
  • #40: Ability to run a Cortex analyzer through the API by Alexander Gödeke
  • #45: Simplify case creation when using a template by providing just its name
  • #49: Add a query builder capability to support TheHive’s DSL query syntax

Paris? Are you There?

Shall you encounter any difficulty, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. As usual, we’ll be more than happy to help!

Zerofox2TH: ZeroFOX Alert Feeder for TheHive

Earlier today, the French (but nonetheless happy) Chefs of TheHive’s code kitchen released DigitalShadows2TH, an alert feeder for TheHive that can consume incidents and intel-incidents from Digital Shadows, a Threat Intelligence provider and feed them as alerts to your favorite Security Incident Response Platform.

We are glad to do the same for ZeroFOX, a social media monitoring platform, with Zerofox2TH. If you are a ZeroFOX customer with a valid API subscription and use TheHive for managing your security incidents and investigating them, you can now feed alerts generated by ZeroFOX to TheHive. Ain’t that joli?

Zerofox2TH is released under an AGPLv3 license (read: free and open source). To use it, you’ll need Python 3, the requests and pillow libraries as well as TheHive4py. You also need TheHive 2.13 or better, with an account on your SIRP that can create alerts.

Please read the README file to learn how to install, configure and run this alert feeder.

Need Help?

Something does not work as expected? No worries, we got you covered. Please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.