Build Your Own Custom Data Provider for Grype with Vunnel

Several weeks ago we announced that we open sourced the process to create a vulnerability database for Grype. A new tool called Vunnel (“vulnerability data funnel”) is the first part of the pipeline. Vunnel takes vulnerability data from an external service like an OS distribution’s vulnerability database or API, transforms it into an intermediary format, and makes it available to Grype-DB. Here’s a sketch of the general architecture:

Grype’s database builder pipeline relies on Vunnel as a key component. Vunnel’s main function is to transform software vulnerability data into a standardized format that other tools can utilize. Vunnel’s Providers, written in Python, are responsible for translating vulnerability information from various sources and formats into a common format.

In this post we’ll walk through an example provider we have written, called “Awesome”, and show how it is put together, and how to build your own. We will assume that you have some Python development knowledge and are at least somewhat familiar with Grype already.

A Quick Tour of a New Provider

First, check out the example “Awesome” provider on GitHub:

README.md

The README has some more details describing how to run the provider in a test environment, some information about code organization, and a few more tips to build a useful and robust provider. To implement your own provider for Vunnel, you will need to implement a class inheriting from vunnel.provider.Provider, and implement two functions: update() and name():

  • name() should return a unique and useful name for your provider. If you’re ingesting vulnerabilities from a Linux distribution, the name of the Linux distribution would be a good choice.
  • update() is responsible for downloading the vulnerability data from an external source and processing it. This is where all of the work is done!

Here is part of our Awesome Provider’s class that implements these two functions (slightly modified for readability):

        # this provider requires the previous state from former runs
        provider.disallow_existing_input_policy(config.runtime)

    @classmethod
    def name(cls) -> str:
        return PROVIDER_NAME

    def update(self, last_updated: datetime.datetime | None) -> tuple[list[str], int]:
        with self.results_writer() as writer:

            for vuln_id, record in self.parser.get():
                vuln_id = vuln_id.lower()

                writer.write(
                    identifier=vuln_id,
                    schema=SCHEMA,
                    payload=record,
                )

        return self.parser.urls, len(writer)

The Provider class has functions to save the processed data in Vunnel’s format, so you don’t need to worry about writing to files or managing storage underneath.

The arguments passed into writer.write include identifier, a unique indicator for a particular vulnerability, schema, the Vunnel schema for the kind of vulnerability you’re parsing (see schema.py for details), and payload, the data associated with the vulnerability:

    def update(self, last_updated: datetime.datetime | None) -> tuple[list[str], int]:
        with self.results_writer() as writer:

            for vuln_id, record in self.parser.get():
                vuln_id = vuln_id.lower()

                writer.write(
                    identifier=vuln_id,
                    schema=SCHEMA,
                    payload=record,
                )

        return self.parser.urls, len(writer)

(from vunnel/blob/main/example/awesome/__init__.py)

As you can see from the example, you may want to factor out the download and processing steps into separate classes or functions for code portability and readability. Our example has most of the parsing logic in parser.py.

In the Awesome example you will find some sections of code labeled “CHANGE ME!”. This is where you will need to make modifications to suit your particular provider.

Trying out the Awesome Provider

To begin, install the basic requirements by following the bootstrapping instructions outlined in Vunnel’s DEVELOPING.md document.

Once you have installed Poetry and bootstrapped the necessary project tooling, you can test the example provider by running:

poetry run python run.py

You should get an output that looks something like this:

tgerla@Timothys-MacBook-Pro example % poetry run python run.py
[DEBUG] config: Config(runtime=RuntimeConfig(on_error=OnErrorConfig(action=fail, retry_count=3, retry_delay=5, input=keep, results=keep), existing_input=keep, existing_results=delete-before-write, result_store=flat-file), request_timeout=125)
[DEBUG] using './data/my-awesome-provider' as workspace
[DEBUG] creating input workspace './data/my-awesome-provider/input'
[DEBUG] creating results workspace './data/my-awesome-provider/results'
[INFO] downloading vulnerability data from https://services.nvd.nist.gov/made-up-location
[DEBUG] clearing existing results
[INFO] wrote 2 entries
[INFO] recording workspace state
[DEBUG] wrote workspace state to ./data/my-awesome-provider/metadata.json

You can inspect the resulting output in ./data/my-awesome-provider/metadata.json:

{
  "schema": "https://raw.githubusercontent.com/anchore/vunnel/main/schema/vulnerability/os/schema-1.0.0.json",
  "identifier": "fake-sa-001",
  "item": {
      "Vulnerability": {
          "Name": "FAKE-SA-001",
          "NamespaceName": "GRYPEOSNAMESPACETHATYOUCHOOSE",
          "Link": "https://someplace.com/FAKE-SA-001",
          "Severity": "Critical",
          "Description": "Bad thing, really bad thing",
          "FixedIn": [
              {
                  "Name": "curl",
                  "VersionFormat": "apk",
                  "NamespaceName": "GRYPEOSNAMESPACETHATYOUCHOOSE",
                  "Version": "2.0"
              }
          ]
      }
  }
}

Now you are ready to modify the example provider to suit your own needs. To contribute your provider to the Vunnel project and share it with the rest of the open source community, you will need to write some tests and create a GitHub pull request. For more information on Vunnel and writing new Providers, you can find a lot more information in Vunnel’s README.md, DEVELOPING.md, and CONTRIBUTING.md documents. Please join us on Slack if you have any questions or need any help. We will be glad to get you started!

The next post in this series will help you connect your new provider to Grype itself. Stay tuned!

How to Check for CISA Catalog of Exploited Vulnerabilities

Last week the United States Cybersecurity and Infrastructure Security Agency (CISA) published a binding operational directive describing a list of security vulnerabilities that all federal agencies are required to fix. Read the directive here: https://cyber.dhs.gov/bod/22-01/ 

The directive establishes a CISA-managed catalog of known exploited vulnerabilities that carry significant risk to federal agencies. The list can be found here: https://www.cisa.gov/known-exploited-vulnerabilities-catalog

While CISA’s directive is binding only on U.S. federal agencies, companies can also leverage this catalog to prioritize vulnerabilities that may put their organization at risk.

There has been a lot of discussion about this directive and what it will mean. Rather than add commentary about the directive itself, let’s discuss what’s actually inside this list of vulnerabilities and what actions you can take to check if you are using any of the software in question.

It’s important to understand that the list of vulnerabilities in this catalog will not be static. CISA has stated in their directive that the list will be modified in the future, meaning that we can expect more vulnerabilities to be added. Even if a federal agency is not currently running any of the vulnerable software versions, as the list grows and evolves and the software that is running evolves, it will be important to have a plan for the future. Think about handling vulnerabilities like delivering the mail. Even if you finish all your work by the end of the day, there will be more tomorrow.

If you work with lists of vulnerabilities you will be used to vulnerabilities having a severity assigned by the National Vulnerability Database (NVD). The NVD is a U.S. government repository of vulnerability data that is managed by the National Institute of Standards and Technology (NIST). The data in NVD enriches the CVE data set with additional product information as well as a severity rating for the vulnerability based on the CVSS scoring system.

It is very common for policy decisions to be made based on the NVD CVSS severity rating. Any vulnerability with a CVSS score of critical or important is expected to be fixed very quickly, while more time is allowed to fix medium and low severity vulnerabilities. The idea is that these severity ratings can help us decide which vulnerabilities are the most dangerous, and those should be fixed right away.

However, this new list of must-fix vulnerabilities from CISA goes beyond just considering the CVSS score. At the time of writing this the CISA list contains 291 vulnerabilities that require special attention. But why these 291 when there are an almost immeasurable number of vulnerabilities in the wild? The directive indicates that these vulnerabilities are being actively exploited, which means there are attackers using these vulnerabilities to break into systems right now.

Not all vulnerabilities are created equally

Examining the catalog of vulnerabilities from CISA, many of the IDs have received a rating of critical or important from NVD, but not all. For example CVE-2019-9978 is a WordPress plugin with a severity of medium. Why would a medium severity rating make this list? Attackers don’t pay attention to severity.

Remember this list isn’t based on the NVD CVSS severity rating, it’s based on which vulnerabilities are being actively exploited. CISA has information that organizations do not and is aware of attackers using these particular vulnerabilities to attack systems. The CVSS rating does not indicate if a vulnerability is being actively attacked, it only scores on potential risk. Just because a vulnerability is rated as medium doesn’t mean it can’t be attacked. The severity only describes the potential risk; low risk does not mean zero risk.

How Anchore can help

There are a few options Anchore provides that can help you handle this list. Anchore has an open source tool called Grype which is capable of scanning containers, archives, and directories for security vulnerabilities. For example, you can use Grype to scan the latest Ubuntu image by running
docker run anchore/grype ubuntu:latest
docker run anchore/grype output
You will have to manually compare the output of Grype to the list from CISA to determine if you are vulnerable to any of the issues, luckily CISA has provided a CSV of all the CVE IDs here:
https://www.cisa.gov/sites/defaultkn/files/csv/known_exploited_vulnerabilities.csv

Here’s a simplified example you can use right now to check if a container is vulnerable to any of the items on the CISA list.

First, use Grype to scan a container image. You can also scan a directory or archive; this example just uses containers because it’s simple. Extract just the CVE IDs, sort them, then store the sorted list in a file called scan_ids.txt in /tmp.

docker run anchore/grype  | sed -r 's/.*(CVE-[0-9]{4}-[0-9]{4,}).*/\1/g' | sort > /tmp/scan_ids.txt

Next download the CISA csv file, extract the CVE IDs, sort it, and store the results in a file called “cisa_ids.txt” in /tmp/

curl https://www.cisa.gov/sites/default/files/csv/known_exploited_vulnerabilities.csv | sed -r 's/.*(CVE-[0-9]{4}-[0-9]{4,}).*/\1/g' | sort > /tmp/cisa_ids.txt

Then compare the two lists, looking for any IDs that are on both lists

comm -1 -2 /tmp/cisa_ids.txt /tmp/scan_ids.txt

The “comm” utility when run with the “-1 -2” flags only returns things it finds in both lists. This command will return the overlap between the vulnerabilities found by Grype and those on the CISA list. If the container doesn’t contain any CVE IDs on the CISA list, then nothing is returned.

Users of Anchore Enterprise can take advantage of a pre-built, curated CISA policy pack that will scan container images and identify any vulnerabilities found that are on the CISA list.

Download the CISA policy pack for Anchore Enterprise here.

Once downloaded, Anchore customers can upload the policy pack to Anchore Enterprise by selecting the Policy Bundles tab as seen below:

Anchore policy tab

Next, upload the policy pack by selecting the Paste Bundle button.

Upload policy bundle to Anchore

If done correctly, you should see something very similar to what is depicted below, where you can see the raw json file loaded into the policy editor:

Loaded policy bundle

Lastly, activate by clicking the radio button for the bundle, so that it can be used in your CI/CD pipelines and/or runtime scans to detect the relevant CVEs from the CISA catalog that are specified within the policy.

Activate a policy on Anchore

You can now see the results generated by the CISA policy pack against any of your images, as demonstrated below against an image that contains Apache Struts vulnerabilities that are included within the CISA vulnerability list.

Policy results

From here, you can easily generate automated reports listing which CVEs from the CISA policy exist within your environments.

Looking ahead

Organizations should expect new vulnerabilities to be added to the CISA catalog in the future. Attackers are always changing tactics, finding new ways to exploit existing vulnerabilities, and finding new vulnerabilities. Security is a moving target and security teams must remain vigilant. Anchore will continue to follow the guidance coming out of organizations such as CISA and enable customers and users to take action to secure their environments based on that guidance.