A modern Sample Exchange System

As an anti-virus software vendor, we exchange current malware samples and IOCs with our partners from the industry. This has been common practice among vendors for many years, but there is no standard for this and the approaches used for the process have specific drawbacks. For example, some vendors provide new malware in zip archives.

1. The malware samples are usually packed into huge archives that contain all the samples collected on one day. You do not know what they contain until you have downloaded and unpacked them.
2. This process can take several hours and is prone to failure. This is precious time lost, which would have been needed to protect our customers.
3. There is a high overlap between individual sample archives, which requires you to spend a lot of time filtering out duplicates of samples you already know.

Our idea is to provide a standardized exchange system which meets the following criteria:

1. The ability to choose only the malware samples you need. Therefore, it is necessary for samples to be named after their SHA256.
2. The ability for researchers to filter samples based on hash, categories and tags before downloading the file. This can include, but is not limited to, the target platform or specific detections.
3. Provide an API that is
   1. Easy to consume and built on current web standards
   2. Easy to set up, so that every exchange partner is able to host it with little added effort

Usually you want to decide which samples to share with which partner. For that it's useful to be able to assign individual samples to sets. A partner then receives a list of samples assigned to their respective set.

Our malware exchange system provides two endpoints.

1. A listing endpoint to request a list of samples for a given time frame, e.g. all new samples from the last hour. The user has to authenticate to receive this list.
2. A download endpoint where the user can request a single sample which he chose from the list.

A GET request against the list endpoint with an authentication header returns a list of Json Web Tokens (JWT), where each JWT contains a sample and meta-information about it. An example JWT from the list looks like this: 

eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzUxMiJ9.eyJleHAiOjE2MDE5NzQ4MjYsInNoYTI1NiI6ImYzZGY5Y2ZiYTIwYzJhNGU1OGNjMzEwNTg2ZTQ5NmYwZTJkZDczMDQwMTZhOTVjMDBhYmU3ZDQ1YTA5MGU4NTkiLCJmaWxlc2l6ZSI6MTA4MjMyLCJwbGF0Zm9ybSI6IldpbjMyIiwicGFydG5lciI6ImNhcm90ZXN0dXNlciJ9.QQYuoqEwK8MyvUKE0GiHCuO1MygRY3znWF5IazYLdmOwCuI1dRJhpvOVJFJgJeGJyRsJ-drijt1VMOkn_yAHHQ

The JWT consists of three parts, separated by a dot. The first part is a header which contains information about the cryptographic algorithm used in the last part of the JWT, which is a signature. The middle part is the base64 encoded payload and can be easily decoded.

{"exp":1601974826, "sha256":"f3df9cfba20c2a4e58cc310586e496f0e2dd7304016a95c00abe7d45a090e859", "filesize":108232, "platform":"Win32", "partner":"username"}

The exp field in the token is an expiration time. A sample cannot be downloaded anymore after the token has expired. In order to download the sample, a new token is required.

The user can now filter the samples based on the hash and the provided meta-information. Based on those criteria, the user can decide which samples are of interest for them. In the second step the user sends the tokens of the samples they are interested in to the second endpoint, which validates the token signature and expiration date. If everything matches, the sample is being downloaded.

An example curl request to download the list of available samples looks like this:

curl -u "YourUserName:YourPassword" 'https://HostNameOfYourExchange.com/v1/list?start=2020-09-23'

To get a sample, send the corresponding JWT:

curl "https://HostNameOfYourExchange.com/v1/download?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzUxMiJ9.eyJleHAiOjE2MDE5NzQ4MjYsInNoYTI1NiI6ImYzZGY5Y2ZiYTIwYzJhNGU1OGNjMzEwNTg2ZTQ5NmYwZTJkZDczMDQwMTZhOTVjMDBhYmU3ZDQ1YTA5MGU4NTkiLCJmaWxlc2l6ZSI6MTA4MjMyLCJwbGF0Zm9ybSI6IldpbjMyIiwicGFydG5lciI6ImNhcm90ZXN0dXNlciJ9.QQYuoqEwK8MyvUKE0GiHCuO1MygRY3znWF5IazYLdmOwCuI1dRJhpvOVJFJgJeGJyRsJ-drijt1VMOkn_yAHHQ"

We open sourced our new malware exchange system under the MIT license on GitHub: Malware Exchange System

We will continue to develop the system on GitHub and hope that other anti-virus vendors or security institutions adopt the system to make malware sample exchange as easy as possible. As next steps, we will improve the documentation and provide more information on the setup of the system.
Feel free to contact us on GitHub if you have questions about the system.