Web Application Scanners Rolling Review Ends - Only One Scanner Was Capable of Scanning the AJAX web Application....You Guessed It - IBM Rational AppScan!

(Subtitle: "We Support AJAX")

Jordan Wiens just concluded the Web Application Scanners Rolling Review, which had been going on for a few months now. This rolling review was very different from past attempts, as it required scanners to be able to scan real-world AJAX web applications, and it was done extremely professionally, while putting an emphasis on scanners' capabilities, rather than on eye-candy features alone.

I must admit that I am very proud today - AppScan, a product that I have been working on for 7 years now, was the only product that managed to scan the application properly, and received great compliments from Jordan.

Here are a few excerpts from AppScan's review, and some final words:

With the exception of IBM's AppScan, automated Web application scanners are simply not yet up to the task of finding security flaws in Ajax code. And it's not like we made it hard on them

 

AppScan's review subtitle:

WatchFire Blazes Past Field.

AppScan was the only product in this Rolling Review to handle Ajax, and it did it without the gotchas that plagued rivals. Now that's hot.

And...

Not only is AppScan the most mature Web application vulnerability scanner on the market, developed in 2000 as a companion to Sanctum's AppShield Web application firewall, it's now owned by one of the most well-known names in computing, IBM, as a result of Big Blue's July acquisition of WatchFire. In the context of this Rolling Review, we weren't sure AppScan's experience would be enough: The Ajax applications we've been feeding our scanners have proved troublesome, even for long-established products. Fortunately for IBM, AppScan looks like a sound investment. It impressed us with its ease of use, advanced functionality and reliability and was the most successful so far at traversing our Ajax applications.

A few words about AppScan eXtensions (and in particular PyScan):

For advanced users, AppScan's built-in utilities are nearly on par with the rich suite of tools integrated into WebInspect. Additionally, since version 7.5, AppScan has taken a cue from the popular Firefox browser by allowing users to develop extensions that can integrate into the product. These add-ons reflect the growing popularity of open-source products and communities. In fact, one sample extension is a complete development environment in itself, integrating the popular open-source scripting language Python with the core engine in AppScan.

Much of the value in a scanner stems not just from how accurate it is, or how flexible its reports are, but from how seamlessly it can be incorporated into your existing workflow to provide meaningful and actionable data throughout the development process. Exposing the product via extensions is a great way to allow customers to use AppScan in a way that best fits their particular needs or environment. Take the Pyscan module. An organization might implement custom scans of different branches of an application under development by automatically scripting both scanning and reporting as code is forked for reuse or checked into a source-code repository. Having the simplicity and popularity of Python tied to the scanning engine that makes AppScan tick is a powerful combination. Creative types will discover a variety of potential uses.

Look, I can sit here, and start lashing back at people who keep saying that automated web application scanners are *pure evil*, but as Master Yoda once said:

Fear is the path to the dark side. Fear leads to anger. Anger leads to hate, and Hate leads to suffering

I think Jordan's Rolling Review proved an important point.... AppScan Rocks!

 

 

** Disclaimer: the author of this blog post does not believe that you can completely secure a web application by only running an automated tool. You can read my thoughts on this subject in my award winning blog post Man vs. Machine

Comments

[Marcin]

"Cenzic Hailstorm and Hewlett-Packard WebInspect (post-update) were capable of analyzing and detecting vulnerabilities in the Ajax application, albeit only when we manually walked them through the relevant bits."

If you don't walk through and set the traversal manually for that, how do you know the scanner is actually spidering successfully?

[AppSecInsider]

I have a different question for you -

How do you know that you (as a human) covered the entire application, when you browse it manually? (consider a VERY large application, containing thousands of web pages, and numerous functionalities)...
=============

A good scanner will give you proper visualization and statistics regarding site structure, pages, HTTP requests, parameters and cookies, so that you can assess (as a human), that it covered the parts of the application that you are interested in testing.

[Romain Gaucher]

Spidering is something, but understanding is also important. How do you know that the tool fully understood the rich content and did a real assessement?

[AppSecInsider]

Hi Romain,

I guess the first question I have, is what does "understanding the application" really means?

does one needs to fully understand the context and functionality of a software in order to be able to test it?

I guess the answer lies in the type of testing you wish to perform. Each type of test requires a different level of analysis and understanding of the application.

Testing for "low hanging fruit" vulnerabilities, which is something that automated scanners are best at, doesn't necessarily mean that you must understand the application like a human does. Sometimes, all you need is to locate the inputs/outputs of the application, and to be able to "understand" or validate the results.

I do agree that certain vulnerabilities (e.g. what people like to call Logical Vulnerabilities), require more human-like attributes, in order to be able to locate (or exploit) them. But, I do believe that automated tools can assist in such scenarios as well (i.e. semi-automatic assessments).

In addition, I do believe that with proper algorithms, heuristics and a bit of innovation, in the future, automated scanners will be able to assist human analysts, more than they do today. Our team has been spending a lot of time in researching such directions, and you can already see such things in AppScan today.

And BTW - did you read my disclaimer? :-)

[Romain Gaucher]

Actually, I was jumping on what Marcin said. How do you know that the tool did the job? It's not because the tool is not reporting anything that you are secure ;)
It's not even because the tool spidered the scripts that it was able to understand it properly...

btw, I did read the disclaimer and I do agree with your post Man vs. Machine

[AppSecInsider]

Lets see if I understand your question - are you asking: "how do I know that the tool located all of the vulnerabilities that exist in the application?"

That's a tough question to answer. Actually, I don't think a human can know for certain that he/she located all of the vulnerabilities, by manually testing the application (even by doing manual code review).

[Romain Gaucher]

Well, I was more talking about understanding rich contents: Flash, JavaScript, etc.

Let's say I have a nice Ajax application, AppScan is able to read it, parse the javascript etc.
How do I know that AppScan understand my javascript properly? If there is some obscure code that it cannot understand (too recursive, totally dynamic,.. ?) will it report it or just don't tell me that it cannot do anything with the script here?

Hope you understand better my thoughts... :X

[AppSecInsider]

Hi,

Assuming you are referring to traversing heavily JavaScripted web applications, then AppScan actually executes the JavaScript code, raises events, and performs actions as if it was a real human user. The output (e.g links, requests, etc.) is then collected and analyzed.

There's no need for AppScan to "understand" your obfuscated JS code. As long as a browser can execute it, AppScan can do the same.

[Jordan]

@Marcin: By monitoring the webserver logs to check for coverage of particular urls after an automated spider.

Also, I did both manual crawls and automated crawls with all products, so it was rather easy to see the differences in findings that way as well.

@Ory: Yup! There are downsides to both methodologies -- I think everybody agrees that the best results require both man and machine to get the job done.

Of course, if all the apps had been java based (only one was), Fortify's Tracer might have been a good way to quantify coverage:

http://www.fortifysoftware.com/products/tracer/

[Romain Gaucher]

So, if, somewhere in my js code, I have an async call which needs a condition which is never reached in my web application, AppScan will not test for that?

function foo(bar = 1) {
if (bar==1) {
// normal use
myxhr('index.py',globalPOST);
}
else {
// this is the only place where 'special/error.py' occurs myxhr('special/error.py', globalPOST);
}

I all my code I only use foo() with no specified parameters. Will AppScan be able to find 'special/error.py' and test it?
If so, I really want to test this -- will need to ask a new version of AppScan before.