The Dockerization Will not be Televised

Do you know what the following set of commands achieve?

$ cd /opt/Cortex-Analyzers
$ sudo git pull
$ for I in $(find /opt/Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip2 install -U -r $I; done \
&& for I in $(find /opt/Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip3 install -U \
-r $I || true; done

The answer is obvious Doctor Watson, right? These highly readable commands (pun intended) allow you to update your Cortex analyzers and responders to the latest stable versions, downloading new ones in the process, going over all the Python 2 and Python 3 dependencies to install the missing ones and upgrade the old ones to make sure they work correctly. These operations take quite a long time and cause some headaches in the process (Hello, I have Python 3.X and this dependency is no longer required, or Hi, I have an old version of Python 2 and it seems I need this other dependency).

And if you are lucky enough to get it running smoothly, you are still not done as you need to log in to the Cortex UI as an organisation administrator (unlike TheHive, Cortex supports multi-tenancy), click on the Refresh analyzers button under Organization > Analyzers then go to Organization > Responders and click on Refresh responders.

So while the answer to the opening question might be simple, updating analyzers and responders is far from being straightforward, to say the least, even if we forget the ugly fact that both are stored in a repository “conveniently” named Cortex-Analyzers*:

thehive@thehive-training:/opt/Cortex-Analyzers$ ls -d a* r*
analyzers  responders

Unnecessary Complexity Must Die

Your lovely, hard-working bees hate unnecessary complexity. Our project’s front page blatantly states our mission to bring Security Incident Response to the masses. And we have to stand by our words even if TheHive and Cortex are free, open source solutions and we do not gain anything from them save for the huge satisfaction of helping our fellow incident handlers level the fight against cybercriminals & all kinds of other animals of the APT (Advanced Persistent ~~Troll~~ Threat) bestiary.

There is only one possible solution: simplify the installation and update process of the current, official 115 analyzers and responders we have as of this writing, the future ones and any private or unofficial ones written in other programming languages such as those developed in Go by Rosetelecom-CERT.

Docker all the Things!

Starting from Cortex 3.0, the next major release of your favourite analysis and active response engine, all analyzers and responders will be dockerized. It will no longer be necessary to install them along with their various dependencies. They will be dowloaded from our cortexengine Docker organisation. Sysadmins might also configure automatic updates.

As a side advantage of using Docker, analyzers, and responders will also be isolated from each other which gives more flexibility and possibilities.

© Steve Simson. This artwork and the title of this blog post are inspired by *The* *Revolution will not be televised*, a song from the late and great Gil Scott-Heron.

For those users who have private, custom analyzers and responders that they don’t want or can’t share with the community, several options will be available:

Continue managing their analyzers and responders in the same way as currently supported by Cortex 2 (i.e. launch them as processes, with no isolation whatsoever).
Dockerize them and store them locally on their Cortex instance.
Dockerize them and publish them on a Docker registry, either the official one or a private registry.

A Docker image of Cortex 3 will still be provided. It will contain a Docker engine to launch dockerized analyzers and responders using DIND (Docker in Docker).

It won’t be necessary to modify the code of the current, official analyzers and responders. A drone job will monitor the analyzer and responder repository and automatically build docker images when it detects changes.

The Cortex Web interface will be slightly modified to accommodate the whole process and allow adding in-house/private Certificate Authorities to allow Cortex to smoothly perform updates in those corporate environments where TLS/SSL inspection is enabled.

Nice Movie Trailer. When is it Coming to a Theatre near me?

We are working hard to get Cortex 3 out of the oven in Q1 (of this year, yes). We will reach out to you, dear reader, in due time, to help us test it and refine it before putting it on the digital shelf for free, as usual. We will provide a smooth migration path in order to move safely your current analyzers and responders and their configuration to Cortex 3.

So to paraphrase the late and great Gil Scott-Heron:

The dockerization will not be televised
The dockerization will not be televised
The dockerization will be live.

Since you are here

The success of TheHive and Cortex continue to grow, far more than we initially foresaw. As far as we know, there are about a hundred organisations of different sizes and locations using or testing them. And as the number of users grows, so does the number of features, professional service and support requests.

We have tried addressing these requests through Creative Source, a nonprofit organisation (NPO). All but one company trusted us enough to make a donation and get tailored services for its needs in return. Most of the others either did not reply to our proposals or explained that their procurement process does not accommodate working with NPOs.

Some members of our core team are actively working on alternative options to ensure not only the viability of TheHive and Cortex as FOSS products on the long run but the ability to provide professional training, support, and services without freaking out highly bureaucratic, think-in-the-box-but-never-outside procurement departments.

Stay tuned 🐝

—
(*) When the idea behind Cortex was born into our hive mind, we did not initially think about active response capabilities. So we naturally called the repository which was supposed to contain analyzers Cortex-Analyzers . When, at a later stage, we added responders, we put them in the same repository for obvious laziness pretences ¯\_(ツ)_/¯.

Correction: February 15, 2019
Typographical errors have been corrected. Some rewording has been made for the sake of clarity.

Cortex 2: a Sneak Peek

Unless you’ve been living in a cave with no Internet connection during the last year or so, you certainly know a thing or two about Cortex, TheHive’s perfect sidekick, which allows you to analyze observables, at scale, using its 30+ analyzers.

As of this writing, the latest version of Cortex is 1.1.4. Cortex can be queried using its Web UI for quick assessment of an observable. But the true power of Cortex is unleashed when the engine is queried through its REST API, either from TheHive (which can leverage multiple Cortex instances), from alternative SIRPs (Security Incident Response Platforms), Threat Intelligence Platforms and programs thanks to Cortex4py. Indeed, when Cortex is called through the API, it can analyze large sets of observables. Each analysis generates a job. Jobs are queued on first-created, first-executed basis.

However, Cortex 1 has three limitations:

It does not support authentication. If you install it and don’t shield it from abuse (using a firewall for example), anyone can submit analysis jobs and consume your query quotas for subscription-based, commercial services, for example. Non-CSIRT/CERT/SOC personnel or threat actors can also view all the jobs you’ve executed (what observables you have analyzed, using which analyzers and what the associated results were).
It does not support rate-limiting. All it takes to ruin your quotas is an unexperienced analyst who’d create a case in TheHive from a MISP event containing thousands of attributes, select them all from the newly created case, and run them through various Cortex analyzers.
It has no persistence. If you restart the Cortex service or the host it runs on, all your analysis results will disappear. Please note that if you query Cortex from TheHive, the latter will keep a copy of all the reports generated by the analyzers.

Moreover, analyzer configuration is not as easy as we’d like it to be. Enters Cortex 2.

Authentication, Organizations, Configuration and Rate Limiting

Cortex 2, due for release in February 2018, almost a year after the release of the first version, will support all the authentication methods TheHive supports: LDAP, Active Directory, local accounts, API keys and/or SSO using X.509 certificates (an experimental feature as of this writing).

Once created, users will be associated to an organization. Each organization has its own configuration: which analyzers are enabled, associated API keys and/or authentication credentials for services (VirusTotal, PassiveTotal, MISP, …) and a query quota.

For example, if you have an overall quota on VT for 10,000 queries/month, you can limit the number of queries to 5000 for org A, 3000 for org B and leave 2000 for other uses. Rate limits can be configured per month or per day.

Screen Shot 2017-12-15 at 17.16.06 — Cortex 2 — Architecture

More on Organizations

Organizations will be ideal for multi-tenant Cortex instances deployed, for a example, by the central CSIRT of a large company. They can then create orgs for their regional SOCs. Commercial teams such as MSSPs will also be able to use a single instance to serve all their customers.

Graphical Interface Enhancements

Administrators will not have to edit /etc/cortex/application.conf by hand to enable and configure analyzers per org. They will be able to do so from the Web UI. The Web UI will also allow them to manage users, orgs and authentication tokens when applicable.

Report Persistence and Freshness

Cortex 2 will use ES 5 for storage, like TheHive. That way, you will no longer lose your existing jobs when you reboot the Cortex host or restart the service. You will also be able to query historical results to monitor changes and so on. We will also add an optional parameter to make Cortex 2 to serve the latest report generated by an analyzer if it is called again, on the same observable in the last X seconds or minutes. That way, we’ll avoid running the same queries again and again for the same observable and thus consuming quotas and CPU and storage resources.

Pricing

Cortex 2 is a significant development over Cortex 1 … but it’ll still cost you nothing as it will remain free and open source. We could feel you itching when you started reading this paragraph. Chill out! But if you are willing to support the project, you can donate to Creative Source, the non-profit organization we have created to sustain TheHive, Cortex and Hippocampe in the long run. Interested? Contact us at support@thehive-project.org then.

Cerana: a Sneak Peek

Initially planned for Nov 17, 2017, Cerana, the next major release of TheHive, is delayed by a few days for three reasons: fixing a few minor but nonetheless irking bugs, quality assurance, and adding small but nice features that would have otherwise required a new database migration a short while after performing one during the upgrade to this new version.

The new release date for Cerana (TheHive 3.0.0) is Dec 5, 2017, the same day we’ll have our second joint workshop with the fine people of the MISP Project during the Botconf conference in Montpellier, France (food, wine, sightseeing… well you get the picture).

If we should mention a single major Cerana feature to convince you to install it or take it for a spin, that would be dynamic dashboards, with no hesitation.

While it was enough for a start, the Statistics module doesn’t take advantage of the underlying Elasticsearch storage and the many ways we can play with all the data that analysts keep feeding to TheHive. Not only that but what about custom fields, alerts, and so on? Enter Dynamic Dashboards.

Dynamic Dashboards – Alert types and sources

To put it simply, Cerana will allow you to analyze TheHive data (almost) any way you want and chart it using different options: how many alerts of a certain type have been received during a given period? Over all the cases that are recorded within TheHive, how many observables with a specific tag and flagged as IOCs are there? …

Dashboards can be private to an analyst, shared with fellow TheHive users, imported from another instance and exported. By adding the import/export feature, we hope to foster sharing within TheHive community where teams would impart useful dashboards to their peers. Graphs can also be saved as images to add to reports.

observable_sources — Dynamic Dashboards – Sources of observables

To alleviate upgrades, Cerana will come with a few dashboards out of the box to mimic the Statistics module hence you won’t lose existing functionality when you make the move. At this stage, we’d like to remind you that we only support the current release and the previous one. When Cerana will be published, we’ll obviously support it (genius, n’est-ce pas ?) as well as Mellifera 2.13.2. Nothing else.

Dynamic Dashboards – Case status, resolution and impact

Cerana will also give you the ability to import and export case templates, a feature that has been requested by our growing user base. This could be a first step towards a global repository where case templates can be shared, refined and created according to common standards, regulations or compliance requirements. Think LPM in France, NIS in Europe, GDPR, etc. Case templates will also be improved to contain default metrics values if needed and automatically assign tasks to given analysts.

Another addition worth mentioning is the sighted flag for IOCs. When an analyst flags an observable as IOC and as sighted, it means that observable is not simply something coming from a sandbox analysis (think C2) or from a 3rd party but was confirmed as being used by a threat actor in your network. In a later release, exporting cases to MISP instances will make use of this new flag to feed MISP attribute sightings. The sighted value will also be used in the future to improve alert previewing.

Last but not least, Cerana will supervise the ‘health’ of the Cortex and MISP instances it is integrated with. The Cortex and MISP logos at the bottom right corner of TheHive UI appear when integration with those products is enabled. They will also have a coloured circle to indicate health:

Green: TheHive can reach all of the configured Cortex/MISP instances.
Orange: TheHive cannot reach all of them.
Red: no instance can be reached.

There are other areas (the About page, the observable analysis buttons…) where the health of Cortex and/or MISP can be monitored.

Now, if you don’t mind, we have some coding to do. We’d better get back to it if we want to give you a luscious release. À bientôt !