« Air Bags by Popular Demand | Main | Looking for Parental Guidance »

August 13, 2007

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d835130c5153ef00e3933b32918834

Listed below are links to weblogs that reference Periodic Blurbs (Warning: Exhortation Inside):

Comments

dre

Maybe I can be objective about this since I try to be heavily involved in the webappsec movement as it is - yet have never worked for a primary company or vendor in that space (well, at least not in many years).

Last year I did some assessments through a few different security assessment companies - all as 1099-misc. Some used web application vulnerability scanners such as SPI, Watchfire, Cenzic, et al. I didn't find any of these tools to be mind-blowingly awesome - and I probably wouldn't buy them if I was in the most likely situation to buy. I'll get to some of the "why" behind this later in this comment.

I've also attended a few OWASP meetings, vendor events, and have tried to remain active on the related forums and blogs, including this one.

Ory - thanks for pointing out this problem because it is a problem. From my perspective (and this may be a result of BH/DefCon) - it is getting worse by the day.

Allow me to summarize the industry war over the last year and what I see happening for the future. I think it's best to understand the problems before we come up with the solutions.

First of all, FortifySoftware has been the primary force trying to sell the idea that SCA scanners find more security related defects than any other solution. By failing to prove this idiom - they designed Tracer (their immature version of a web application fault-injection scanner) along with AspectJ bytecode weaving and EMMA (a code coverage tool) to show that dynamic-only fault-injection scanners do very little testing of the overall application.

But SCA scanners don't do testing - they do inspection. These are developer tools, and they are much cheaper than dedicating more resources (people, process, time) to peer review. If you can put a $2k/dev SCA security scanning tool into the hands of all your developers along with an advanced code coverage tool such as the $250/dev Clover tool (from Cenqua, now Atlassian) - then you'll be able to demonstrate using security metrics (see: Jaquith) that this method improves application security and is certainly worth the money invested.

Could you use PMD and EMMA instead? The simple answer is yes, but Fortify SCA and Clover are much more advanced [today] and include support contracts. Depending on how your business operates - you'll have to figure out which best suits your needs.

Secondly, web application fault-injection scanners have had to come up with new models to even be sold properly ($2k for a pen-tester 2-week license, $15k for a one year SaaS license, integration with operations, QA, and dev tools, etc) because of customer needs. This has hurt their business tremendously. Instead of concentrating on selling their tools to smart people who know they need it - these companies are starting to sell tools to fools.

Gadi Evron already established in "Fuzzing the Corporate World" that the only people that buy fuzz testing tools today are security product vendors (fuzzing before release) and financial companies (fuzzing before purchase). You should know your market before you start spending a lot of company money trying to "create a market" after you spent too much money on the product and a sales team of dead ringers.

Vulnerability sportsmen are also fighting to stay alive. Do you know why? Because they add zero value to the development process. They are all ego-driven by their work. Have you ever heard of a vulnerability hunter talking about how they discovered a vulnerability? Talk about full-disclosure! Have you ever heard of one including a coverage report with code complexity metrics with their findings? Do they ever talk about which taxonomy of errors they used - or which heuristics they created? Do they say if they know it is only repeatable under certain conditions because it requires fuzz stepping? Do they indicate where in the threat-model of the application their vulnerability findings fits? No, not a single person has ever done this - and it's not included as features in the fault-injection scanners available today. It's all Finding #1,2,3 - High/Med/Low.

Vulnerability hunters will include such information in their vulnerability reports, especially going forward. The market for binary and bytecode analysis of code coverage is bound to be larger than the SCA scanner market - not because "companies won't give out their source" - but instead because it actually is a viable way to find vulnerabilities (as opposed to SCA security scanners which find defects). However, just like PMD could not replace Fortify SCA - FindBugs or FxCop also have no chance of replacing what Veracode does today - not that any of these are bad tools. It's all about who uses them and for what reasons. PMD, FindBugs - et al - are for finding all types of code level defects, not just security vulnerabilities.

So, in summary - static source code security scanners, and static file security analyzers will be marketable solutions to all web application developers across the world for the development inspection process.

Hybrid fault-injection scanners will be marketable solutions to QA and security professionals (in dev/QA/ops/3rd-party) - and the instructional capital behind these hybrid scanners (and their reports) will be marketable only to verticals that require this level of assurance (e.g. financial, high-end e-commerce, and maybe healthcare). However, more and more companies will be requiring this sort of acceptance, operations, and maintenance testing between each other (also see: PCI DSS, OWASP secure software contract annexes, etc).

Lastly, and almost not worth mentioning - is WAF and APIDS. I have always contended the uselessness of these solutions because I don't even understand the problem that they are trying to solve. FortifySoftware Defender and Determina Memory Firewall aside - these products sometimes prevent some scenarios for some application vulnerabilities under certain trivial circumstances. So they're a great one-week $25k answer to a $2.5M problem you're facing. You are certainly better off with a 2-week pen-test using a dynamic-only fault-injection scanner and you'll never hear that from me again.

So, again, Ory - good post. In the last paragraph you provide the viable solution that nobody wants to listen to. Don't you vendors know what's going on in the real world? Have you ever been a developer for a mission-critical web application? Have you ever been an operator (yes, I guess security engineering for Yahoo or eBay like RSnake and Jeremiah did count - but they are obviously too immature, and also can't remember any lessons learned: probably blinded by the dollar signs they are seeking, or possibly jealously of not being bought by big-named computer companies like some of their competitors)?

I think Andrew van der Stock has been saying similar things recently (1) especially concerning the problems with spending too much time on FUD - and not enough time on solutions.
(1) http://www.greebo.net/2007/07/23/final-score-oscon-4234-black-hat-592-defcon-1118-appsecurity-10444-statistically-insignificant/

AppSecInsider

"Last year I did some assessments through a few different security assessment companies - all as 1099-misc. Some used web application vulnerability scanners such as SPI, Watchfire, Cenzic, et al. I didn't find any of these tools to be mind-blowingly awesome - and I probably wouldn't buy them if I was in the most likely situation to buy. I'll get to some of the "why" behind this later in this comment."

>>>>
Fair enough – although I bet if you gave me 30 minutes of your time, I'll be able to "convert" you to the dark side :-)
>>>>

"First of all, FortifySoftware has been the primary force trying to sell the idea that SCA scanners find more security related defects than any other solution. By failing to prove this idiom - they designed Tracer (their immature version of a web application fault-injection scanner) along with AspectJ bytecode weaving and EMMA (a code coverage tool) to show that dynamic-only fault-injection scanners do very little testing of the overall application."

>>>>
I don't think this was the beginning of it all – there was a lot of anti-blackbox/anti-automation going on long before Fortify even existed. Actually, I think Fortify Tracer is a brilliant idea.
You see, blackbox scanners are not "robohackers", often times they require configuration and customization in order to be able to properly scan an application. Using Fortify Tracer (or any other code coverage tool), one can know what parts of the application where not scanned, and go back to tweak and configure the blackbox scanner to work properly on the application.
Having said that – I do think that Fortify's Tracer sales approach was a bit aggressive, and they took the easy path – pushing their own technology, while trashing the blackbox vendors. I actually mentioned this to Brian Chess during the OWASP Conf. in Milan.
I know that my own company (and probably other blackbox vendors) never trashed source code analysis as a technology. We never saw the need to advance our own technology on the expense of others. There are enough security issues for everyone!
>>>>

"Lastly, and almost not worth mentioning - is WAF and APIDS. I have always contended the uselessness of these solutions because I don't even understand the problem that they are trying to solve."

>>>>
You don't understand what WAFs are supposed to do? Come on…. I've seen some great WAFs around (especially Sanctum's AppShield RIP).
>>>>

Anyway dre, thanks for taking the time to read my long post, and posting your own long comment :-)

dre

@ Ory,

You said, "Fair enough – although I bet if you gave me 30 minutes of your time, I'll be able to `convert' you to the dark side".

Actually, it's funny you mention that. In the past two years, the only cold-call that I've received as a result of filling out whitepapers was from Watchfire. This was around November or December of last year. I talked with the sales guy for about an hour - he was highly entertaining and knew the subject-matter well. I told him that I didn't think [web app scanners] were mature enough to consider now, but that they probably will be in 2007.

Then you said, "I don't think this was the beginning of it all – there was a lot of anti-blackbox/anti-automation going on long before Fortify even existed".

Oh very true. It's just the starting point in our conversation that I chose to begin with. It's also one of the more popular arguments (SCA vs. Fuzz testing) on various mailing-lists (e.g. fuzzing, secure coding, et al).

You also said, "We never saw the need to advance our own technology on the expense of others. There are enough security issues for everyone!"

Not everyone sees it that way. I think the problems go much deeper than some of the issues we both hit on.

Lastly, you said, "You don't understand what WAFs are supposed to do? Come on. I've seen some great WAFs around (especially Sanctum's AppShield RIP)".

Sure, I mentioned Fortify Defender and Determina Memory Firewall... OWASP Stinger and others are certainly in this category. I guess I was referring to vendors such as F5, Cisco, et al. I certainly do understand the problem they are trying to solve - I'm just not certain that they solve it.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Follow us on Twitter

AppScan Free Trial


Try IBM Security AppScan software at no charge.

Become a Fan