Cortex 2: a Sneak Peek

Unless you’ve been living in a cave with no Internet connection during the last year or so, you certainly know a thing or two about Cortex, TheHive’s perfect sidekick, which allows you to analyze observables, at scale, using its 30+ analyzers.

As of this writing, the latest version of Cortex is 1.1.4. Cortex can be queried using its Web UI for quick assessment of an observable. But the true power of Cortex is unleashed when the engine is queried through its REST API, either from TheHive (which can leverage multiple Cortex instances), from alternative SIRPs (Security Incident Response Platforms), Threat Intelligence Platforms and programs thanks to Cortex4py. Indeed, when Cortex is called through the API, it can analyze large sets of observables. Each analysis generates a job. Jobs are queued on first-created, first-executed basis.

However, Cortex 1 has three limitations:

  1. It does not support authentication. If you install it and don’t shield it from abuse (using a firewall for example), anyone can submit analysis jobs and consume your query quotas for subscription-based, commercial services, for example. Non-CSIRT/CERT/SOC personnel or threat actors can also view all the jobs you’ve executed (what observables you have analyzed, using which analyzers and what the associated results were).
  2. It does not support rate-limiting. All it takes to ruin your quotas is an unexperienced analyst who’d create a case in TheHive from a MISP event containing thousands of attributes, select them all from the newly created case, and run them through various Cortex analyzers.
  3. It has no persistence. If you restart the Cortex service or the host it runs on, all your analysis results will disappear. Please note that if you query Cortex from TheHive, the latter will keep a copy of all the reports generated by the analyzers.

Moreover, analyzer configuration is not as easy as we’d like it to be. Enters Cortex 2.

Authentication, Organizations, Configuration and Rate Limiting

Cortex 2, due for release in February 2018, almost a year after the release of the first version, will support all the authentication methods TheHive supports: LDAP, Active Directory, local accounts, API keys and/or SSO using X.509 certificates (an experimental feature as of this writing).

Once created, users will be associated to an organization. Each organization has its own configuration: which analyzers are enabled, associated API keys and/or authentication credentials for services (VirusTotal, PassiveTotal, MISP, …) and a query quota.

For example, if you have an overall quota on VT for 10,000 queries/month, you can limit the number of queries to 5000 for org A, 3000 for org B and leave 2000 for other uses. Rate limits can be configured per month or per day.

Screen Shot 2017-12-15 at 17.16.06
Cortex 2 — Architecture

More on Organizations

Organizations will be ideal for multi-tenant Cortex instances deployed, for a example, by the central CSIRT of a large company. They can then create orgs for their regional SOCs. Commercial teams such as MSSPs will also be able to use a single instance to serve all their customers.

Graphical Interface Enhancements

Administrators will not have to edit /etc/cortex/application.conf by hand to enable and configure analyzers per org. They will be able to do so from the Web UI. The Web UI will also allow them to manage users, orgs and authentication tokens when applicable.

Report Persistence and Freshness

Cortex 2 will use ES 5 for storage, like TheHive. That way, you will no longer lose your existing jobs when you reboot the Cortex host or restart the service. You will also be able to query historical results to monitor changes and so on. We will also add an optional parameter to make Cortex 2 to serve the latest report generated by an analyzer if it is called again, on the same observable in the last X seconds or minutes. That way, we’ll avoid running the same queries again and again for the same observable and thus consuming quotas and CPU and storage resources.

Pricing

Cortex 2 is a significant development over Cortex 1 … but it’ll still cost you nothing as it will remain free and open source. We could feel you itching when you started reading this paragraph. Chill out! But if you are willing to support the project, you can donate to Creative Source, the non-profit organization we have created to sustain TheHive, Cortex and Hippocampe in the long run. Interested? Contact us at support@thehive-project.org then.

Featured

Introducing Cerana

Update: 2 days after publishing this blog post, we’ve released Cerana 0.1 (TheHive 3.0.1) which fixes a number of issues. We encourage you to use 3.0.1 instead of 3.0.0.

The friendly honeybees at TheHive’s code kitchen were pretty busy lately even though winter came and temperatures have been close to zero Celsius in Paris, France. As we wrote a couple of weeks ago on this very blog, we are happy to announce Cerana to the world, available immediately.

Cerana or TheHive 3.0.0 is the latest (and obviously greatest) release of a now highly popular open source, free Security Incident Response Platform (or SIRP for short). Its flagship feature in comparison to previous releases is Dynamic Dashboards.

Dynamic Dashboards

Dynamic Dashboards replace the Statistics module in Cerana to allow you to explore the data available in Elasticsearch, which TheHive uses for storage, in many ways. For example, you can have a usage breakdown of Cortex analyzers, the number of open cases per assignee, the number of alerts per source (MISP, email notifications, DigitalShadows, Zerofox, Splunk, …), the number of observables that have been flagged as IOCs in a given time period, how many attributes were imported from MISP instances, top 10 tags of imported MISP attributes or incident categories.

case3.png
Dynamic Dashboards

Dynamic Dashboards can be created by an analyst and kept private or shared with the other team members. Dashboards can also be exported and imported into another instance. This would facilitate community participation in the establishment of valuable data exploration graphs to drive DFIR activity and seek continuous improvement.

When you’ll migrate to Cerana, you won’t have to build dashboards from scratch. We recreated more or less those which were available under the Statistics view and included them in the Cerana build.

Cortex and MISP Health Status

Cerana will also allow you to monitor the health status of all the Cortex and MISP instances that it is connected to. In the bottom right corner of TheHive’s Web UI, the Cortex and MISP logos appear when you have configured the integration with those products as in previous releases. However, the logos will have a small outer circle which color will change depending on whether Cortex and/or MISP instances are reachable or not.

status
Cortex & MISP Health

If TheHive can’t reach N out of M Cortex/MISP instances, the outer circle will be orange. If it can’t reach all M instances, the circle will red. If everything is fine, the circle will be green. The exact status of each Cortex/MISP instance can be seen in the About page. And when you try to run analyzers on a Cortex which cannot be reached, TheHive will tell you so as well.

about
Cortex & MISP: Version & Status

Sighted IOCs

In previous releases of TheHive, observables can be flagged as IOCs. However, this doesn’t necessarily mean you’ve seen them in your network. Think for example of a suspicious attachment which you’ve submitted to Cuckoo or Joe Sandbox through Cortex. The analyzer returns some C2 addresses to which the sample tries to connect to. You’d be right to add those C2 addresses to your case and flag them as IOCs. Then you search for them in your proxy logs and you find connection attempts to one out of four. In previous versions, you’d add a seen label but this would be inconsistent among analysts. One may use found instead. Another will add a description and no labels.

To avoid such situations and give you a simple way to declare an IOC as seen, Cerana adds a sighted toggle which you can switch on/off. We will leverage this toggle in future versions to indicate sightings when sharing back cases to MISP.

Other Features and Improvements

Cerana contains numerous other features and improvements such as:

  • Case template import, export
  • The ability to assign default values to metrics and custom fields to case templates
  •  The ability to assign by default tasks to their rightful owners in case templates
  • Show already known observables when previewing MISP events in the Alerts page
  • Add autonomous systems to the list of default datatypes
  • Single-sign on using X.509 certificates (in BETA currently)

We will update the documentation for Cerana in the upcoming weeks. So stay tuned.

Download & Get Down to Work

If you have an existing installation of TheHive, please follow the migration guide.

If you are performing a fresh installation, read the installation guide corresponding to your needs and enjoy. Please note that you can install TheHive using an RPM or DEB package, use Docker, install it from a binary or build it from sources.

Support

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.

TheHive, Cortex and MISP: How They All Fit Together

TheHive, Cortex and MISP work nicely together and if you’ve read our June-Dec 17 roadmap post, the integration of our products with the de facto threat sharing platform will get better in a few months.

During the FIRST conference presentation we gave last week, we displayed a picture that we will use here to try to explain how these three open source and free products integrate with one another.

Screen Shot 2017-06-16 at 09.58.43.png
A Picture is Worth a Thousand Words…

TheHive

TheHive is a Security Incident Response Platform (SIRP). It can receive alerts from different sources (SIEM, IDS, email. etc.) via its REST API. This is where alert feeders come into play.

Alert Feeders

Think of an alert feeder as a specialized program which consumes a security event (SIEM alert, email report, IDS alert, and so on), parses it and outputs an alert that its sends to TheHive through TheHive4py, the Python library we provide to interact with TheHive’s REST API.

We do not supply such feeders but developing them should be straightforward. If not, let us know  and we’ll do our best to help you out.

Alerts

Any alert sent to TheHive will show up in its Alerts pane. In addition to the sources mentioned above, new or updated MISP events will show up as well in that area if you configured TheHive to connect to one or several MISP instances. If so, TheHive will poll those MISP instance(s) at every interval looking for new or updated events. If there are any, TheHive will generate an alert which will end up in the Alerts pane.

Screen Shot 2017-06-18 at 15.29.51.png
The Alerts Pane

Alerts can be ignored, mark as read, previewed and imported. When an alert is imported, it becomes a case that needs to be investigated.

Cases

workflow
The Workflow that is at the Heart of TheHive

A case can be generated from an alert or created from scratch. It is subdivided into tasks (think identification, containment, eradication, check proxy logs, and so on) and observables (IP addresses, hashes, email addresses, domain names, URLs…). When analysts are working on tasks, they add logs as they go. In TheHive’s terminology, logs are text entries which may contain attachments to help analysts record what they have been doing. Logs can be written using Markdown or a rich-text editor.

Case Templates

You don’t need to add the same tasks over and over when working on cases belonging to a given category (DDoS, Malspam, APT, …). You can create custom templates to which you add tasks as shown below. This is very useful when you are dealing with alerts so that when you import them, you can select which case template you’d like to apply and there you go!

Screen Shot 2017-06-16 at 10.26.22.png
A Sample Case Template
Observables

Observables can be tagged, flagged as IOCs, and analyzed. When the investigation is well in progress or completed, you may want to share the resulting IOCs or a subset of those with partners and peers. TheHive will support the ability to export that data to MISP in September 2017. Until then, you can still export your IOCs as text, CSV or as a MISP-compatible format that you can use to add them to your MISP instance using the freetext editor. TheHive can export IOCs/observables in protected (hxxps://www[.]somewhere[.]com/) or unprotected mode.

Every observable must have a TLP (Traffic Light Protocol) level. By default, any added observable is considered TLP:AMBER. Please note that the TLP is taken into account by some analyzers. Wait! Analyzers?

Cortex

Cortex is our standalone analysis engine and a perfect companion for TheHive and MISP. Analysts can use it to analyze observables using its Web UI, in which case they can be submitted only one at a time. The Web UI should really be limited to quick assessments of observables before creating a case in TheHive (or in an alternate SIRP). The power of Cortex really comes into play when you use its REST API. TheHive speaks natively to Cortex (as MISP does). Moreover, TheHive can leverage one or several Cortex servers.

Screen Shot 2017-06-18 at 15.57.27.png
Observable Page and List of Analyzers
Analyzers

As of this writing, Cortex has 23 analyzers which come in a total of 39 flavors and more will be available soon.

An analyzer can be written in any programming language supported by Linux though all of our current analyzers are written in Python. This is because we provide a Python library called Cortexutils which contains a set of utility classes that make it easier to write an analyzer in Python.

Flavors

Analyzers such as VirusTotal, PassiveTotal or DomainTools can provide different analysis services. Let’s take VirusTotal as an example. You can scan a file or URL. That’s one flavor. You can also obtain the latest available report on VirusTotal.com for a file, hash, domain or IP address. That’s a second flavor. So the VirusTotal analyzer has two flavors.

Screen Shot 2017-06-18 at 16.26.41.png

How about PassiveTotal? It has 8 flavors: unique resolutions lookup, SSL certificate history lookup, malware lookup, passive DNS lookup, data enrichment lookup, SSL certificate details lookup, OSINT lookup and WHOIS data lookup.

The MISP Search Analyzer

At this point, we need to mention a special analyzer that may create some confusion if not understood correctly: the MISP Search analyzer. Thanks to it, Cortex has the ability to search observables within a MISP instance as represented by the arrow that goes from the Analyzers to MISP.

Screen_Shot_2017-06-19_at_08_03_54.png
Search for MISP Events Containing a Given Observable

When an observable is found in an event, Cortex will return the number of records found (i.e. the number of events where the observable has been found) and a list of links to those events with additional data.

Screen_Shot_2017-06-19_at_08_13_16.png
Searching for a Hash Using the MISP Search Analyzer from the Cortex Web UI
Screen Shot 2017-06-19 at 08.17.04.png
The Same Search Conducted from TheHive: Long Report
Screen Shot 2017-06-19 at 08.18.58.png
Mini-Report

The current version of the MISP Search analyzer can only search within a single MISP instance but in the near future, it will be able to support multiple ones.

MISP Expansion Modules

Besides its own analyzers (which include MISP Search described above), Cortex can also invoke MISP expansion modules. These are normally used by MISP to enrich attributes within events but Cortex can also take advantage of them to analyze observables.

There is some overlap between the native Cortex analyzers and MISP expansion modules. When choosing between a native analyzer or an expansion module, we highly recommend you select the former. The expansion modules are deactivated in the default Cortex configuration.

Jobs

When you submit an observable for analysis, Cortex will create a job and, if successful, it will generate an analysis report in JSON format. TheHive has the ability to parse those results and present them in a human-friendly fashion thanks to report templates we offer for free. So when you’ll submit an observable to Cortex from TheHive, you’ll get back a short (or mini) report and a long one. The first can be thought of as a really tiny Exec Analyst Summary while the second provides more insight and details.

Calling Cortex from MISP

In addition to the expansion modules we have just mentioned, MISP 2.4.73 and up can enrich attributes using Cortex analyzers. The configuration is pretty straightforward. So if all you are concerned about is threat intelligence and sharing, you may augment your visibility into a given threat represented as a MISP event by leveraging all current 23 Cortex analyzers and any future ones.

Conclusion

TheHive, Cortex and MISP are three open source and free products that can highly aid you combat threats and keep the ‘monsters’ at bay.

TheHive, as a SIRP, allows you to investigate security incident swiftly in a collaborative manner. Several analysts can work simultaneously on tasks & cases . While cases can be created from scratch, TheHive can receive alerts from different sources thanks to alert feeders which consume security events generated by multiple sources and feed them into TheHive using TheHive4py Python library. TheHive can also sync to one or several MISP instances to receive new and updated events which will appear in the alert pane with all the other alerts generated by other sources. Analysts can then preview new alerts to decide whether they need to be acted upon. If so, they can transform them into investigation cases using templates.

To analyze the observables collected in the course of an investigation and/or imported from a MISP event, TheHive can rely on one or several Cortex analysis engines. Cortex is another standalone product that we have developed which sole purpose is to allow you to analyze observables at scale thanks to its large number of analyzers, MISP expansion modules and any analyzer you might have developed on the side. Cortex has a REST API that can be used to empower other security products such as  ‘analytics’ software, alternate SIRPs or MISP.

The highly popular threat sharing platform can indeed enrich attributes thanks to Cortex as it has a native integration with it. And in a few months, you will also be able to export cases from TheHive as MISP events that you can share with peers and partners.

If you do share, you do care about our collective mission to defend the  digital assets that are under our watch from harm. So let us fight together as one.

 

 

TheHive: June-Dec 17 Roadmap

A new version of TheHive will be released by the end of June. We will take this opportunity to review our release naming and numbering from the ground up.

Months ago, we started giving ‘major’ versions (2.10, 2.11, …) the name of honey bee varieties. 2.10 was called Buckfast. 2.11, the current version, is called Mellifera. And we were supposed to give 2.12 yet another name. However, and after the few hiccups we’ve encountered with our QA as of late, we have decided to change things around in order to make sure new releases are as stable and well-maintained as you should expect them to be.

Starting from the next release (2.12), we will abide by the following numbering scheme:

  • Major versions == X (2, 3, …)
  • Minor versions = X.Y (2.12, 2.13, 3.1, …)
  • Hotfix/maintenance versions = X.Y.Z (2.12.1, 2.13.2, 3.1.1, …)

Only major versions will have corresponding honey bee names. So long as we stay with v2, we’ll keep calling all the minor versions Mellifera N (2.12.0 = Mellifera 12). Version 3 will be called Cerana.

Mellifera 12 – June 29, 2017 (planned date)

Mellifera 12 (v 2.12) will succeed to Mellifera 2 (the current version) to comply with the new naming scheme. It will allow you to see how similar new alerts are to existing cases so you can decide whether you import them into an existing case, create a new one or ignore them altogether. Mellifera 12 will show you the status of all the related cases (#229) to the one you are working on. Finally, you’ll have the ability to change the default case template before importing an alert.

M12 will also support custom fields (#12), a feature that has been requested by numerous users. This version will also add mini-reports to the Observables tab. That way, once a Cortex analysis has been completed, analysts will be able to view part or all the resulting short report in that tab instead of having to navigate to the page of each observable to read the short report.

Mellifera 13 – September 14, 2017

TheHive 2.13 should be the last Mellifera version. It will complete TheHive’s integration with MISP by adding the ability to export all observables or a subset of them to a MISP instance. Please note that TheHive allowed you from the start to import events from multiple MISP instances but since sharing is caring, we wanted to add the ability to export to this very popular threat sharing platform from your Security Incident Response Platform (SIRP). We do not want to rush it though.

Cerana – October 12, 2017

Cerana or TheHive 3.0.0 will bring a complete UI overhaul to make it even easier to work on cases, perform analysis and get your job done, after the interface refreshments Mellifera brought. It will lay the ground for some nifty features we have in mind.

Cerana 1 – November 15, 2017

TheHive 3.1.0 will include dynamic dashboards: the ability to work with the statistics and metrics the way you want and generate customized dashboards to help you drive your activities.

Keep an eye on TheHive’s milestones on GitHub. There are other features and enhancements that we might add as we progress and we will reflect them on that page.

Correction: June 12, 2017
An earlier version mentioned GitHub issue #36 as pertaining to custom fields while it is a request for globally-defined tags that an analyst can choose from.

 

 

 

Mellifera Is Here

TheHive Project French chefs are very excited to announce the immediate availability of Mellifera, TheHive 2.11.0, the greatest and latest iteration of our flagship product.

We are thrilled to share this major version with the incident response community, for free as usual. Yes, you read that sentence right. You don’t have to cough up a single € or BTC for a platform that is as good as some commercial alternatives, unless your boss is hassling you about paying big bucks to get so-called professional support. If that’s the case, try us and you might prove them wrong.

Going through all the features and fixes of this significant overhaul will take forever (well, almost) so let us highlight a few that we feel worthy of your attention and time.

The Alerting Framework

If you need one reason to upgrade from Buckfast to Mellifera or to ditch your existing, clunky incident handling platform and use ours, then that should be its brand-new and powerful alerting framework.

With Buckfast (TheHive 2.10.x) and earlier versions, you can configure multiple MISP instances. TheHive will then poll those instances at regular intervals and display new or updated events in a specific area where analysts can preview them, import them as cases using configurable templates or ignore them altogether (and if they do so by mistake, there’s no way to go back). And if you needed to raise alerts from a SIEM, email reports or other sources of noteworthy security events, you had to rely on TheHive4py API client and create a case without having a chance to preview the events in TheHive prior to the case creation.

Mellifera does not have these limitations. It features an all new, fancy and efficient alerting framework which can be displayed using the Alerts button in the Web interface. This button was previously called MISP.

Within the Alerts area, you can preview not only new or updated MISP events but also any event that you have pushed through TheHive4py. The client has been modified to be compatible with Mellifera. If you have an existing TheHive4py package, please upgrade to the new 1.2.0 version using PIP.

sc-thehive-alerting-filters.png
The New Alerting Panel

Using TheHive4py 1.2.0, you can send your SIEM alerts, user email reports and security events from various sources to Mellifera and your analysts will be able to preview and import them or simply ignore them. If they have ignored some events by mistake, they can use the quick actions on the top of the panel to retrieve them. Please note that you have to create programs that will bridge your event sources with Mellifera through TheHive4py.

sc-thehive-alerting-stats.png
Stats within the Alerting Panel

All New Skin

Mellifera has an all new skin with many refinements spread all over the interface. For example, you can now easily reorder the tasks within a case template. You can also sort task logs according to their creation date (oldest first, newest first). The flow (a.k.a live stream) is also collapsible. Moreover, when you create a case, Mellifera will suggest existing tags.

sc-thehive-main.png
Mellifera’s Brand New Skin

Is MISP or Cortex There?

If you have configured Mellifera to interact with at least one MISP or Cortex instance, the Web interface will show their respective logos at the bottom of the page. Please note that you can now connect to MISP and Cortex even if you are behind a proxy which requires authentication.

sc-thehive-mispenable.png

sc-thehive-cortexenable.png

New Installation Packages

Starting from this release, we no longer produce all-in-one binary packages and dockers containing TheHive and Cortex. Instead you can use dockers, binaries and RPM as well as DEB packages. Wink wink.

One More Thing

Mellifera has an all new logo and the project website has been completely redesigned. Now you can see who’s behind the project thanks to Alexandre Gohier, a close friend who also happens to be a professional photographer.

Download & Try

If you have an existing TheHive installation, please follow the new migration guide.

If you are performing a fresh installation, read the installation guide corresponding to your needs and enjoy!

Support

Something does not work as expected? You have troubles installing or upgrading? No worries, please join our  user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We are here to help.