Handling Complex Scenarios with AppScan's Custom Parameters

If you are familiar with web technologies, either from assessing them for security, or from developing them, you are probably aware of the many innovative ways developers choose to implement web applications. The days of following the HTTP RFC are long gone - developers do whatever they need to do, in order to make things work.

For example, HTTP Parameters are passed as a part of the URL (e.g. REST), tokens are passed as HTTP headers (e.g. CSRF protection) instead of as cookies or parameters, parameter values are concatenated with weird strings instead of ampersand (&), XML islands inside HTTP request body, JSON, etc.

The task of writing a good, robust and thorough web crawler is almost impossible these days - unless you take into consideration the fact that by the time you finish writing your parser, you'll have to add support for new technologies and new web development paradigms.

What I'm trying to say is that if you're trying to write a good web crawler, you better find a way to enhance it on the fly, and do so in a simple manner, so your users will be able to figure it out.

That's exactly what we've done with AppScan Standard's "Custom Parameters" definition. This capability, which I have already blogged about when I discussed testing of RESTful services with AppScan Standard, allows AppScan Standard users to augment AppScan's HTTP parser capabilities, in almost endless ways. 

The "Custom Parameters" definition, enables an AppScan user to teach AppScan how to extract non-standard parameter structures from HTTP traffic. In addition, once you define such a parameter, you can apply a set of rules specifically for that parameter (or parameter group) - for example, should it be tested? should it be tracked? from the login sequence, or whenever a new value is set by the application, and so forth.

The definition itself is done using regular expressions, which are extremely powerful. You can define where the parameter is located in the HTTP request, if it has a parameter-name, or perhaps only a value, how parameter name/value pairs are concatenated, etc. - it is truly the ultimate way to extend AppScan's HTTP parser to support the most bizarre scenarios.

And why am I suddenly bringing this up?

A few days ago, there was an interesting discussion going on over at Twitter (originally raised by Dan Cornell from the Denim Group), and in a few blogs, about a complex login scenario, which AppScan supposedly required help from Burp proxy. At some point, it was decided that AppScan failed. In reality, all that was needed from AppScan to support that complex login, was a simple "Custom Parameter" configuration.

Once I submitted the sample configuration back to Dan (which was contributed by our own Paul Ionescu - a true AppScan expert), he kindly posted the solution on his blog, and the situation now is that AppScan is currently the only scanner with a validated solution.