Joe Sandbox, MISP Search and Report Improvements

We are thrilled to announce that Cortex has two new analyzers: Joe Sandbox and MISP Search. Moreover, we have produced new analyzer report templates for TheHive and improved existing ones.

Joe Sandbox

List JSB Cortex.png — Cortex: New Joe Sandbox Analyzer

Joe Sandbox, by Joe Security LLC, is a very powerful malware analysis platform that has been around for many years and comes in two flavors: cloud and on-premises. The Joe Sandbox Cortex analyzer has been tested using an on-prem Joe Sandbox Ultimate version and can process URLs and files. The analyzer can process files with or without Internet access.

To use the analyzer, you must provide the API key of your Joe Sandbox instance. You must log in to Joe Sandbox, click on your account name, then on Settings and on the API Key tab.

report JSB Cortex.png — Cortex: Joe Sandbox Output Example

We have produced a report template for the Joe Sandbox analyzer output resulting from file analysis. The URL analysis report template is not yet available but it should be in a few days.

TheHive: Joe Sandbox Analyzer – Short and Long Report Samples

MISP Search

Screen Shot 2017-03-31 at 15.25.07.png — Cortex: New MISP Search Analyzer

It is no longer necessary to present MISP, the de facto standard of threat sharing. The new MISP Search analyzer will let you search events containing the observable you provide as an input. It applies to a lot of observable types as you can see in the screenshot above.

To use it, you’ll need to supply the API key available in the MISP UI interface.

Nils Kuhnert created an alternate MISP Search analyzer which has the ability to query multiple MISP instances. We are currently reviewing his submission along with several other analyzers he contributed before improving the newly released MISP Search analyzer.

PassiveTotal Report Templates

While we published the PassiveTotal analyzer weeks ago, TheHive didn’t have report templates for it at the time. We have now new, shiny short and long report templates for most of the services provided by the PT analyzer.

PT PDNS long report.png — TheHive: PassiveTotal PassiveDNS – Long Report Sample

DomainTools Whois Lookup Report Template

The short report templates of the DomainTools Whois Lookup analyzer has been improved. We now use a taxonomy to provide more context and differentiate between the DomainTools and PassiveTotal Whois results.

VirusTotal Get Report and VirusTotal Scan Report Templates

The short report templates for both services have also been improved to use a taxonomy to provide additional context and distinguish their results from the PassiveTotal Malware service.

Get the new analyzers

To install the new analyzers, grab the Cortex-Analyzers repository and unpack its content (or git pull the master branch) in your existing /path/to/cortex-analyzers.

The Joe Sandbox analyzer does not need any additional Python library if you have already installed Cortex and the analyzers following the guide we provide. To use it, edit your Cortex configuration file (/path/to/cortex/application.conf) and add the following lines in the analyzer section:

 JoeSandbox {
     apikey="..."
     url="..."
 }

By default, Joe Sandbox will time out the analysis after 30*60 seconds (30 minutes). Additionally, the analyzer will wait for the Joe Sandbox server to respond within 30 seconds. If no response is received within this period, it will time out. If you want to override these values, you’ll need to add the following lines in the analyzer section:

JoeSandbox {
     apikey="..."
     url="..."
     analysistimeout=<NEW VALUE> # optional
     networktimeout=<NEW VALUE> # optional
}

The MISP Search analyzer requires pymisp. Use the following command line to install the required library:

sudo pip install pymisp

Then edit your Cortex configuration file (/path/to/cortex/application.conf) and add the following lines in the analyzer section:

MISP {
     api_key="..."
     url="..."
}

Please note that you must restart Cortex to take the changes into account. The current version has no persistence so you’ll lose all your existing jobs.

You can find the full installation requirements for Cortex and Cortex-Analyzers on the Cortex wiki pages.

Use the New Report Templates

To import the new report templates in your instance of TheHive:

download the updated package
log in TheHive using an administrator account
go to Admin > Report templates menu
click on Import templates button and select the downloaded package

Running Into Trouble?

Shall you encounter any difficulty, please join our user forum, contact us on Gitter, or send us an email at support@thehive-project.org. We will be more than happy to help you!

Jigsaw Falling Into Place

While we released TheHive as a free, open source product in November 2016, it must not be chalked off quickly as a young, immature solution.

v1.0.0 was put into production in our environment in October 2014. Yes, October 2014. And we’ve been using it every day and refining it since then. Once we deemed it good enough, we decided to share it with the community under an AGPL license to help incident responders in their mission.

Make no mistake. TheHive is a field-tested, mature Security Incident Response Platform (SIRP) built by people who are passionate about Digital Forensics and Incident Response.

A few months after the first public release (v 2.9.0), we adopted bee-related codenames for new major versions and published Buckfast (v 2.10.0).Cortex, the analysis engine that allowed TheHive to analyze and assess observables at scale was shipped as a separate product. Buckfast can interface with one or several Cortex instances depending on your performance and OPSEC needs. For example, you may want to install a separate Cortex on your investigation, air-gapped network to interact with your sandbox as you don’t want to be firing those malicious samples on your corporate network.

Buckfast can also create cases out of MISP events. You can configure it to import them from a single or many MISP instances. And to prepare for the next major version, Mellifera, due in early May 2017, we have released TheHive4py, a Python API client for TheHive.

TheHive4py will be improved to fully support Mellifera’s alerting framework. To put it simply, Mellifera will not only let you preview MISP events and import them but also receive SIEM alerts, email incident reports and different other types of alerts depending on your environment thanks to TheHive4py. And if an analyst discards an alert by mistake in Mellifera’s notification area, they can go back to a ‘trash bin’ and fix their error. Mellifera will also allow you to export cases as MISP events to share IOCs with other teams.

TLP-WHITE-Jigsaw_Falling_Into_Place-2017-03.001.png — Jigsaw Falling Into Place

Now lets’ get back to TheHive’s perfect companion: Cortex. As of this writing, Cortex features 13 analyzers. These analyzers can perform one type of analysis (such as Abuse Finder) or several (such as DomainTools which can do 6). In the very near future, we plan to add at least 10 more analyzers which are shown in the boxes with dotted borders in the picture above. All upcoming analyzers are contributed by our user community whom we wholeheartedly thank. One of the analyzers will allow you to check observables from TheHive against a MISP instance to search for events that may contain them.

We have also begun work on a Python API client for Cortex dubbed… Cortex4py (how creative wink wink). This will allow people who are not using TheHive to summon the power of Cortex from their SIRP, scripts or any other DFIR tool that can import or interact with Python code.

So in the few months since our project was born to the Internet, we have released a solid collaborative SIRP, a simple yet powerful analysis engine to analyze observables and aid teams in their investigations as well as a Python API client for our SIRP. We also have rather ambitious plans to make them even much more useful.

Oh and one more thing! We have released another piece of software around the same time as the first version of TheHive and on which we haven’t said much so far: Hippocampe. Hippocampe can regularly download feeds and exposes a REST API to let you query them from Cortex (or from other tools). You submit an observable and it’ll tell you if it appears in one or several feeds along with a score. The score takes into account the trust you put in the feed sources (which can be adjusted over time) and the number of sources which contain the observable. We’ll cover Hippocampe in more details in an upcoming post.

Before you run away from us
Before you’re lost between the notes
The beat goes round and round
Jigsaw is falling into place
So there is nothing to explain

Buckfast 1 and Cortex All-in-one Package

When you use TheHive, running an analyzer on an observable through Cortex will generate a long report and, in most cases, a short report as well.

Let’s see how this works in practice through an example. Assume we are trying to assess whether the 636a4249104acaaf6d76d7409dc3cb2d MD5 hash is malicious or not:

Screen Shot 2017-03-26 at 22.21.10.png

We start by clicking on it, which will open a new tab: Screen Shot 2017-03-26 at 22.21.22.png This TLP:WHITE hash was imported from a MISP event published by our good friends at CIRCL.lu sometimes ago. As you can see from the screenshot above, no analyzer was executed on it. Let’s check if it is known to VirusTotal (VT). To do so, we just need to click on the fire icon located at the right side of the VirusTotal_GetReport_2_0 row.

Screen Shot 2017-03-26 at 22.21.53

A blink of an eye later, the job has finished successfully as we can tell from the green checkmark. Clicking on the date will let us see the long report, presented according to a report template that we freely provide with most analyzers to the exception of PassiveTotal (but in a few days, PT will also get its own nifty templates). Screen Shot 2017-03-26 at 22.22.11.png

Since we are checking whether VT knows a hash or not, it will give us the results if any corresponding to the last time the associated file was scanned on the service. In our case, this dates back to Dec 2, 2016.

When the analyzer was executed, it also produced a short report which TheHive displays below the observable: Screen Shot 2017-03-26 at 22.21.45.png Short reports come in 4 colors. Red means danger (what else?). Orange means suspicious. Green means innocuous. And blue is informational. OK but what does this have to do with the title of this post?

A few days ago, while working on a new set of analyzers, Nils Kuhnert reported an issue in Buckfast 1 (2.10.1) pertaining to short reports on observables. When he ran some analyzers that should have produced short reports, he didn’t get any. When he reverted to Buckfast 0 (2.10.0), it worked. We tracked down the problem and found that our build process was the culprit. The all-in-one binary package which was supposed to contain Buckfast 1 and Cortex was in fact a 2.10.0 TheHive snapshot that had a regression. We have uploaded a fresh all-in-one binary package with Buckfast 1 instead of the development snapshot.

If you have grabbed the binary all-in-one package, please download it again and update your instance. If you are using a docker version or built Buckfast 1 from sources, you are fine. To make sure you are running the right version, click on your username once you are logged in then on About TheHive. You should see the following information: Screen Shot 2017-03-26 at 22.56.09.png

We are going to review our release process from the ground up to ascertain such errors never occur again. We expect it to be ready for Mellifera, our next major release of TheHive. Please note that starting from that release, we will no longer provide all-in-one binary and docker packages. Instead, we’ll have separate packages for TheHive and Cortex. TheHive4py and the upcoming Cortex4py will be made available through PIP.

TheHive 2.9.1: a Case Merging Example

As this reporter informed you a few weeks ago, the fine cooks of TheHive Project were working hard on a new version of TheHive (TH) which most prominent feature is case merging. Put simply, starting from 2.9.1, TH lets you merge two cases together once you realize that they are similar to each other if not outright identical. This will save you from tearing your hair up and duplicating work on two separate investigations only to realize down the DFIR road that they should have been one right from the start.

When you perform a merge, the two original cases will be used to create a new one that is a union of all the observables of its parents. However, all the tasks of the parents will appear in the child, even if a task has the exact same name in the original cases. This might look as a shortcoming but pause a second and think about it. Tasks are basically worklogs and if we were to fuse a task, how would we collect the worklogs in each of them and put them back in an intelligible form? Got any effective ideas? Then what are you waiting for? Shoot them our way.

In the meantime, let’s see how this works in practice through an example. Earlier today*, Antoine, our intrepid security incident response lead of the StarGazer CSIRT imported a MISP event that looked interesting (to him at least):

2-9-1-release-misp_event_import — Antoine imports a Malspam-related MISP event

While Antoine can import a MISP event as a TH case that has no associated tasks and create those by hand, he prefers automating as many things as possible. So he previously logged in as an administrator, then he created a case template for importing MISP events. The template contains the typical set of minimal tasks that need to be performed during the investigation of such events. He hasn’t supplied numerical metrics for the template at this time.

Meanwhile, Isabelle, his fellow analyst, was monitoring the CSIRT mailbox for user reports and notifications that may need to be acted upon. As it happens, Anthony Braco from the HR department forwarded a suspicious email which resembles a malspam:

2-9-1-release-new_case_isa — Isabelle creates a new case upon receiving Anthony’s report

She extracts the suspicious attachment, fires it up in the team’s sandbox and extracts a number of observables which she starts adding to the case:

2-9-1-release-new_observable — Isabelle adds a URL

Interestingly, the URL she has just added has the eye icon which means it has already been seen somewhere else. She clicks on the observable to get the details:

2-9-1-release-links — The URL has already been added to another case

Under the Observable Links section, Isabelle observes that the URL was added to the case #10 that has been created a few minutes ago. She executes the URL Category analyzer that was kindly contributed by Eric Capuano and integrated to TH 2.9.1. The website is flagged as malicious:

2-9-1-release-urlcat — URL Category shows that the URL is malicious

Isabelles then clicks on the Case Summary tab and makes sure that her case shares the observable with the one that was created by Antoine:

2_9_1_release-shared_cases — The Related cases section is of interest

sabelle then decides to perform a merge of her case with Antoine’s. This will let the team avoid duplicating efforts and leverage the attributes from the MISP event that Antoine converted speed up the investigation and cover more ground during the identification, containment and eradication phases of incident response. To do that, she clicks on the facing arrows icon:

2_9_1_release-merge-step1 — Isabelle searches for the case to merge with hers by number

2_9_1_release-merge-step2 — Isabelle selects case #10 and clicks on Merge

The two cases are then merged and a new one created, containing all the observables, analyzer results, cases and tags of the originals:

2_9_1_release-merge-done — A new case is born

Once the merge operation is completed, the original cases are closed:

2_9_1_release-merge_dup — View of one of the original cases

Besides the case merge feature and the addition of the URL Category analyzer, 2.9.1 fixes a number of bugs and adds a few enhancements, many of which were brought to our attention by our user community. Please read the full changelog for additional details.

If you are a current 2.9.0 user, we highly recommend you update to 2.9.1.

* Please do not put too close an attention to the dates in some of the cases which are off by several weeks. We are intentionally using old data for the sake of demonstrating the feature.

Case Merging

The chefs behind TheHive authorized this reporter a sneak preview into their code kitchen as they were preparing a delicious recipe for an upcoming release: case merging.

Reading about someone else’s experience with food, wine, music or in this case code can be baffling but let us not shy away from a tedious task and call an example to the rescue.

Antoine is the security incident response lead of the StarGazer CSIRT which is using TheHive (TH) to handle incidents and keep the monsters at bay. Their TH instance is connected with their MISP server and Antoine has been keeping an eye on the top navigation bar of the application to spot new MISP event notifications that would need processing. Here comes one. Antoine clicks on the notification bubble and is taken as a result to the Import MISP events view.

misp_events-anonymized — The Import MISP events View

Indeed, a new MISP event showed up from a partner’s instance that is synced with StarGazer’s. Antoine previews the event, decides it worths an investigation and creates a TH case out of it with a number of associated tasks by clicking on import event. StarGazer CSIRT has indeed taken advantage of TH case template engine to match handling MISP events to their processes.

misp_preview-anonymized — Preview a MISP event before importing it

Isabelle, a StarGazer security analyst, has just wrapped up a task from a previous case, looking for suspicious activities in proxy logs. She completed her analysis, updated the task in TH and closed it. Since the Flow was open on her screen, she saw Antoine creates the new case and the related tasks. As it happens, one of them consists of searching URLs copied over from the MISP event in the proxy logs.

Isabelle clicks on the new task, an action that automatically assigns it to her, and starts working on it. Antoine sees what Isabelle is up to thanks to the Flow and then moves on to deal with a different task.

A few minutes later, Antoine’s eyes are caught by a new MISP event notification. He can’t handle it right now so he reaches out to Sabine, the backup incident response lead, and asks her to take care of it. Sabine obliges and previews the new event which is from a different source than the first one. She decides that it deserves an investigation as well and there, she creates a new case. Time flies by and StarGazer analysts are buzzing along the two investigations only to realize that the two MISP events were in fact almost the same.

Here is what happened. The two events were published by two different sources who were subjected to a spear-phishing attack by the same threat actor. While the malicious attachments are unique for each recipient, they drop the same malware which beacons using HTTP to a small set of C2s that were successfully identified by both parties. The email addresses used by the attacker are different but not the MTA. Moreover, the email subjects have the same pattern.

In this situation, using the current TH version (2.9.0), the analysts’ hands are kind of tied. They may close the second case after manually copying the observables that were created out of the new MISP event to the case opened by Antoine for example. What a hassle. It kinds of defeat the productivity boost they were expecting by choosing TH.

Enters case merging.

In TH version 2.9.1, which will be released in a few days, the StarGazer team will be able to merge both cases very simply, creating a new one, without losing observables, tasks or their associated logs. So even if a new MISP event pops up hours or days later, they will be able to create a case out of it and merge the case right away with an existing one if similarities are found. And spotting similarities is quite easy with TH since when you create and update a case, the Web UI Case View will tell you if there are look-alike cases based on observables. You can also navigate to the observables tab of the new case, choose to display 100 at a time and see if most observables have an ‘eye’ icon associated with them (which means they have already been seen). Of course, this second method is not ideal and if you want be thorough you’d want to review if all the observables have an ‘eye’ icon.

Case merging will not be limited to cases created out of MISP events. You may use this feature for any case that you have created in TH.

But what happens if, after merging, the MISP event that was used to create one of the original cases is updated and new attributes have been added to it? As this reporter was pestering the chefs with questions, they asked him to leave the kitchen since they were putting the very last touches to the recipe so he complied.

Well, all we have to do is grab a nice table, put a napkin on our lap, and wait for the new version to be served. Maybe then our question will be answered.