CWE (Pie Chart) is in the Eye of the Beholder

A few days ago, I was sitting down at my desk, doing my daily blog reading session, when I stumbled upon a post at the Matasano Chargen blog. The post (which quotes Mark Curphey) mentions a presentation that Jeff Williams gave at a New York OWASP chapter meeting, in which Jeff uses the following slide:

Jeff took some information that was posted by MITRE, and used it to demonstrate that if you take all automated security tools, and use them all together against something, the tools will only cover 45% of the known vulnerability types enumerated by the MITRE CWE project. 

Since I was an active participant in the CWE project (on behalf of Watchfire), I know a thing or two about the CWE project. Specifically, I know that:

1. The CWE project includes input from many different types of sources, such as Source Code Scanning vendors, Web Application Scanning vendors and all sorts of preliminary taxonomies. This super-set-input, includes numerous entries that are not what you would call a vulnerability per-se.
2. If you look at the CWE project as a tree of 600+ vulnerability types, there are 104 (non-leaf) nodes, which describe categories, and not vulnerabilities.
3. Many of the CWE vulnerability types, are design issues, or business logic issues.

So, using the CWE information as-is, without digging deep into it, and understanding its structure and nature, will get you to the same conclusion, that automated tools only cover 45% of the CWE vulnerability types, while in reality, things are not as simple and bleak as they seem (check out the CWE full dictionary page, and you'll understand why I say it is not simple)

However, I do agree (and I did state this in this blog before) that automated tools have a lot of room for improvement, and that some things cannot be fully automated, but I certainly can't say (based on the current statistics) that they cover 45%, 65%, 75% or any other percentage of the vulnerabilities in the world.

In addition, I do believe that the CWE project is super-important, and will supply us with great statistics (that's why I was happy to participate in the project), which end users will be able to use for their benefit (evaluating tools, building proper testing processes), and tool vendors will be able to use for improving their products.

To wrap things up, you can read MITRE's official response to the whole CWE Pie Chart evolution, which they kindly posted on the Matasano Chargen blog, after I've raised my concerns to them about the validity of the pie chart that was circulating in the blogosphere.