IBM Application Security Insider

Testing RESTful Services with AppScan Standard

As much as I love SOAP web services (not!), it seems like RESTful web services really caught on and became a de-facto standard these days – you see them everywhere, in the cloud, in AJAX or Web 2.0 applications, mobile applications and so forth.

Unlike SOAP services, RESTful services are lightweight. They are extremely easy to understand and also to develop. Nevertheless, there seem to be a million different definitions as to what they really are, but I think the simplest way to understand them is by using the following four definitions, which I’ve found in this DeveloperWorks article:

1. RESTful services use HTTP methods explicitly
2. RESTful services are stateless
3. RESTful services expose directory structure-like URIs
4. RESTful services transfer XML, JSON or both

Simple right?

As much as RESTful services are simple for humans to understand, they are actually a nightmare for automated web application scanners. Why? Because classic HTTP requests usually include parameters either in the Query or Body part of the request. On the other hand, RESTful services usually pass them as what looks like directories (see rule #3 above). For example – the following HTTP request will return the details for a user named Bob:

GET /data/users/Bob/ HTTP/1.1

Host: www.some.site

Connection: close

 

If this was a standard HTTP request, I would tell you that there’s a good chance you’re looking at a web server that contains 3 directories under its virtual root /data/, /users/, and /Bob/, but that’s not the case. This request, tells the RESTful service to retrieve (GET) the account information for user Bob, which is a part of the /users/ list in our data repository.

When an automated scanner crawls the web application, there’s a good chance that out-of-the-box, it won’t figure out that we’re looking at a RESTful service here, and it will consider these parts of the URL as directories. This means a few things:

• Directory-level tests will be sent to the wrong places – potential false positives
• Parameter-level tests will not be sent to the right places – potential false negatives

IBM's AppScan Standard enables you to train it to cope with RESTful services, using one of two options – Manual or Automatic configuration. Let’s start with the manual option.

Custom Parameters

By default, AppScan automatically recognizes parameters in standard HTTP & HTML formats, but if parameters are in other formats (for example within the Path or within another parameter), you need to define them manually, so that AppScan would be able to recognize, follow and manipulate them during scanning. This is done from the Custom Parameters definition, which you can find under Scan Configuration -> Parameters and Cookies -> Advanced: Customer Parameters.

In order to create a new type of custom parameter definition, you have to click the “+” button, which opens the following screen:

 Let’s see a step by step process of adding a definition that will properly parse and test our “users” parameter in the example above.

1. We’ll start by giving this custom parameter definition the Reference Name RESTful_Path_Parameter
2. In the Pattern field, we’ll enter the regular expression /data/([\d\w\s%]+)/([\d\w\s%]+)/ - This pattern includes two match groups, i.e. /data/group1/group2/, group1 denotes the parameter’s name, and group2 the parameter’s value
3. Since the name of the parameter is the first match group, we will define the Name group index as “1”, and since the value of the parameter is the second match group, we will define the Value group index as “2”. This tells AppScan to extract the name of the parameter from group1, and the value of the parameter from group2. If you are dealing with a Path that only includes a parameter value (i.e. nameless parameters), you can set the Name group index to an empty value, and only mark a single value group
4. Our RESTful service uses Path based parameters, so we’ll set the Location to “Path”. In general, you can set it to either “Body”, “Path”, or “Query”.
5. In our scenario, we’ll leave the Condition Pattern empty. This pattern helps us to limit the behavior of the custom parameter definition, by setting another pattern match on the Location. For example, we could’ve defined the Condition pattern to be: ^/data/, and then our pattern parameter definition would only be relevant for Paths that actually begin with /data/.
6. In addition, in our scenario, we will leave the Response Pattern empty. Just as an FYI - this pattern helps us to teach AppScan how to track the values of our custom parameter in scenarios where the application treats it as a session ID. In such cases, the application might not only embed new values in Paths (e.g. in web links), but also in other places in subsequent responses, such as XML elements, for example: <newSessionID>12345678</newSessionID> - In this case, we would have defined the following Response Pattern:
   <newSessionID>([0-9]+)</newSessionID> - this tells AppScan that even though in the HTTP request, the parameter is called users, it should extract new values from an XML element in subsequent responses, that is called newSessionID. Tricky, complex but nevertheless useful!

That’s it. Once we have our custom parameter definition in place, we can let AppScan crawl and test the application normally. After the Explore phase, you can have a peek in the Data view, and look at the Script Parameters table:

As you can see above, each new RESTful parameter that is extracted and analyzed by AppScan is given a special name in the following format:

__patternParameter__[REFERENCE_NAME]_[NAME_GROUP]__INDEX

In our case, AppScan detected the users parameter with 2 values – Bob and Jane, and the books parameter with two values Bobs Biography and Janes Biography.

The INDEX part of the custom parameter is helpful if the regular expression that we created, caught on the same Path more than once. For example, consider the following Path:

/data/users/Bob/data/phone/areacode/

 our pattern would actually match twice on this Path - the first match (index = 0) would set the parameter name to be users and its value to be Bob, and the second match (index = 1) would set the parameter name to be phone and the its value to be areacode. In such case, the name of the custom parameter would appear as:

__patternParameter__RESTful_Path_Parameter__phone__1

Explore Optimization Module

Mastering AppScan’s custom parameters definition could be a daunting task, but this feature is extremely powerful and will allow you to create complex definitions that could parse non-standard HTTP messages of any type and form. If you are in a hurry, lazy, or simply hate regular expressions, there’s an automated way to detect custom parameters by using AppScan’s Explore Optimization Module, which is available through the Tools->Extensions->Explore Optimization Module (Configure or Run):

This extension runs a smart algorithm that will statistically detect URL rewriting rules, such as those that are heavily used by RESTful web applications to generate its directory structure-like URLs. For example, given enough URLs of the format /data/users/VALUE/... this module will automatically generate a custom parameter definition for you.

How much is “enough URLs”? This depends on the configuration of the module and specifically on its Switch Complexity Limit, which by default is set to 50, meaning that you must have 50 different values for the /users parameter.

If you want this module to automatically kick in during scans, you can enable it by going to: Tools->Extensions->Explore Optimization Module: Configure, and checking the box next to Always run automatically during scans. The module will start working once AppScan has crawled 1,000 URLs. You can increase or decrease this default threshold through the Minimum links to start module configuration. If you suspect that your application is using RESTful services, and the module was disabled when you first scanned it, you can always simply run it by going to:  Tools->Extensions->Explore Optimization Module: Run

After the module ran, AppScan’s scan log will include special messages related to this module, for example:

There you go. All I had to do was to let AppScan crawl the application for a few minutes, then Run the module, and it automatically created a custom parameter definition with the regular expression users/([^/]+)

In general, the more URLs you have, the better this module will behave.

It is also iterative - if you continue scanning the application after it created the first round of definitions, and once it hit the threshold again, or once you clicked on Run, it will refine these rules and create new ones where needed. Simple and elegant, albeit less accurate and powerful than the manual option mentioned earlier. That's it.

This post was a bit long, you probably need a REST now.

Through the Looking-Glass

 In recent years, I've heard many industry luminaries lament the untimely death of black box web application security assessment tools. Most of them did so, on the basis that white box tools (static analysis code scanners) will eventually rule the world and bring order to our galaxy.

The top three rants against black box testing tools were usually –

1. Application coverage (or lack thereof)
2. Scan results inaccuracies (false positives, negatives, and non-exploitable issues)
3. Lack of code-level information for issues reported (mainly, vulnerable line of code)

Regardless of these problems, many still consider black box testing tools to be more accurate, practical and mature tools. Issues reported by black box scanners are usually real and exploitable, and they are less prone to generate noisy results than white box testing tools (taint analysis covers all code paths, which can be a double-edged sword)

Our team has been working extremely hard to remediate these shortcomings, and find creative ways to make automated black box testing more efficient, accurate and to make our customers more successful in securing their web applications.  Our latest innovation was dubbed Glass box testing. Actually, it's not such a new idea - this is something that we have been toying around with for many years (we eventually filed the patent back in Feb. 2008)

Finally, it is here.

One of the main drawbacks of black box testing is the fact that the tester (or tool) is completely oblivious to the inner workings of the tested application it is facing. It is unaware of the programming language, the operating system, database server and so forth. The entire process of issue validation in black box testing relies solely on how the application reacts to certain HTTP requests. For example - you send a SQL injection test and then expect to see a SQL syntax error sent back in a subsequent response. Sadly, many of the most critical issues may never reflect back in some scenarios.

Glass box testing, avoids the classic black box pitfalls, by deploying a server-side agent, which gleans critical application information in runtime, and sends it back to the black box scanner. The type of information relies on the type of agent being used, and the location at which it operates, within the application or near it.

For example, imagine a glass box agent, which performs code instrumentation to the tested web application, it sits in critical points and monitors certain events. When a black box scanner sends a SQL injection test, that will reach a sensitive method (sink), our glass box agent will be able to notify the scanner that a vulnerability exists [Problem #2 solved]. Moreover, it will be able to report back on the vulnerable file name, line number, enclosing class and method, as well as the library class and method that was vulnerable –e.g. Java's executeQuery [Problem #3 solved]

 In order to solve application coverage issues, various different kinds of glass box agents can be used, the simplest would be an agent that monitors the web application's file system, and submits back information about its structure and files. A run time code instrumentation agent similar to the one mentioned earlier, could also detect certain unreferenced parameters - those that are mentioned in code, but never referred to by HTML - like 'debug' parameters [problem #1 solved]. There are many other things a glass box agent could do to improve and complement black box testing, this is merely the tip of the iceberg.

Up until now, I discussed the merits of glass box testing in general. Now I would like to briefly discuss the unique capabilities we've built into AppScan Standard 8.5 that was released yesterday.

AppScan’s first glass box agent, performs runtime analysis through code instrumentation of web applications. It hooks into the application, and observes critical application points, reporting vital information back to AppScan Standard. This information includes:

• Monitoring of sensitive methods (sinks) for a range of application layers attacks, bringing unparalleled ultra high accuracy to existing issue detection, as well as the capability to detect issues that black box scanners simply can't spot
• Detecting unreferenced application parameters, and increasing application crawling coverage dramatically

Other than the cool detection capabilities that were implemented, we had to make sure our users are successful in deploying and using glass box. We can't simply build a half-baked prototype, right? So we baked into it some nifty and important features such as:

• Simple agent installation, through an automatic installer for Windows, Linux and Unix
• Manual agent installation & step-by-step guide for complex systems and environments
• Agent authentication & support for SSL communications with AppScan
• Agent command & control, which allows AppScan to remotely push new (digitally signed) logic & rules into deployed agents
• Reduced agent footprint. When the agent is not being actively used, it stops monitoring the application to avoid even the minimal effect on performance
• Real time agent status monitoring. From as early as the scan configuration phase, and through the entire scan, AppScan users know exactly what is going on with the agent. Is it up, down, monitoring or configured improperly
• Agent logging. Glass box logs can be remotely configured and accessed from AppScan's user interface
• Agent management. A cross-scan persistent management of known agents, so you won't have to configure the same things over and over again.

 As you can see, we take glass box testing very seriously. This is not just a new feature in the product.

We believe glass box is the future of dynamic analysis.

We continuously benchmark our glass box testing capabilities on many different web applications, trying to assess the exact increase in scan accuracy and coverage. In reality, it really depends on the specific application that is being tested. But when we ran AppScan Standard 8.5 with and without glass box, on a sample set of the standard known vulnerable web applications, we found that it brings an increase of at least 180% in real true findings, and removes false positives completely.

Glass box testing provides the best of both white and black box worlds. It enjoys the merits of dynamic analysis - the fact that it is testing a real live web application, and provides real exploitable issues without any noisy ‘theoretical’ results. And at the same time, it enjoys code-level information, similar to static analysis - the ability to look at the application's code and inner workings, and tie each issue to the specific location where the defect lies.

In fact, this reminds me that glass box testing is actually not our first attempt to marry white box and black box – our previous product release, included the highly successful groundbreaking JavaScript Security Analyzer, which uses a hybrid dynamic and static analysis engine, to analyze JavaScript code for client-side vulnerabilities. Back when it was released last year, we reported that JSA found JavaScript issues in 14.5% of the Fortune 500 web sites – in AppScan Standard 8.5, we actually went back and improved our analysis algorithms, and we now know that JSA finds security issues in 40% of the fortune 500 web sites (as reported in the latest 2011 IBM X-Force trend analysis report). That is an incredibly awesome yet disturbing piece of data.

In order to give our customers the best possible way to secure their applications, we are continuously blurring the lines between the different types of analysis approaches. I personally think that the entire black box vs. white box debate is way past us. Security testers shouldn’t really care what analysis they are using, as long as it produces accurate and actionable results.

• Glass box testing is available as a feature of AppScan Standard Edition 8.5, and does not require any integration with other IBM products. Simply download and install AppScan Standard 8.5
• Glass box testing is a patent pending technology – US patent application 20090205047
• AppScan Standard customers can download the latest release, which includes glass box testing from this link

Proactive Malware Scanning

I'll start with a short personal angle -

I have a friend that works as a freelance web site developer and webmaster. Once in a few weeks he gives me a call, telling me that one of the sites he manages seems to be serving malicious JavaScript code to its users. It appears to me that this problem is getting out of hand these days, sites are getting (silently) hacked into, and JavaScript code is injected and later on served to users.

From what I hear and read, more than 70% of the Malware today is being served or linked from legitimate web sites.

Take a look at this article from InformationWeek, which was posted in January 2009:

Seventy percent of the top 100 Web sites either hosted malicious content or contained a link designed to redirect site visitors to a malicious Web site during the second half of 2008

The common approach to Malware protection and Malware scanning today, puts the (security) responsibility on the end users (browser protections, A/V, etc.) or the organizations (content filtering gateways, A/V gateways) from which the end users browse the web from.

I think that web site owners should start taking responsibility for the contents they are serving to users, and a simple way to do that, is to constantly monitor or scan your own web application for malicious contents.

About two years ago, I had an interesting thought - if you are already scanning your web application with an automated scanner, that has the capability to perform deep crawling and analysis (using automatic form filling, JavaScript and Flash execution, etc.), why not attempt to locate malicious code that is being served to your web users?!

BTW, malicious code can end up in your application in different ways such as -

• Someone hacked into your application and put it there
• You are including web contents (or application code) from a 3rd party. This is oftentimes the case in Web 2.0 scenarios
• You pissed off one of your web developers, and they decided to get back at you by infecting your users with Malware

Enter Malware Scanner AppScan eXtension

The Malware Scanner AppScan eXtension helps you verify that your application is not hosting or linking to malware. The extension couples the deep-scanning capabilities of IBM Rational AppScan with ISS X-Force technology that is used to identify malicious content and links.

The Malware Scanner checks these conditions:

• Files hosted on your application are malicious or not
• Files that are "one click" away from your application are malicious or not
• Links on your site lead to malicious domains (malware sites or phishing sites, for example)
• Links on your site lead to unwanted content (illegal sites, hate sites, adult content, and so forth).

The Malware Scanner works in two phases:

1. It passes all of the visited links through the ISS Virus Prevention System (VPS) engine, to determine whether they are malicious or not. This is similar to browsing every page in your application, including clicking every button and downloading every file, using a machine with updated antivirus software.
2. It passes all of the links that lead to external domains through the ISS WebFilter SDK. This SDK then fetches the classification of each link (news site, porn site, malware site, illegal site, and so forth), based on the constantly updated online classification database. Links that are deemed malicious or unwanted are flagged for your attention.

When something needs to be brought to your attention, a security issue is created in Rational AppScan so that you can benefit from the strength of Rational AppScan results management capabilities, such as creating reports, saving and loading scans, and so forth.

You can read more about the Malware Scanner eXtension and download it from our eXtensions web site (you need to have AppScan installed to run it).

There's a New AppScan In Town

I usually don't tend to blog about our product releases, but yesterday we have launched the official new version of IBM Rational AppScan Standard Edition (version 7.8), which includes some capabilities that I believe are worth blogging about.

Here's a short list of the interesting new features and capabilities:

• Flash execution & Testing: AppScan now automatically crawls Flash applications to reveal web application vulnerabilities, including vulnerabilities unique to Flash such as XSS in Flash, Phishing through Flash (Redirections), Cross Site Flashing, Insecure Direct Object Reference, Over permissive Flash Sandbox, Over permissive crossdomain.xml files
• AMF Parsing & Testing: On the same subject of Flash testing, AppScan is now capable of parsing and analyzing AMF communications between Flash applications and their back-end server side application.
• Content-based Application Mapping: many modern web applications (especially those designed with the MVC paradigm) make use of a single URL, and serve contents based upon different parameters. In such scenarios, it is irrelevant to report vulnerabilities based on URLs. AppScan 7.8 allows you to create or modify the application tree by defining a criteria by which AppScan will assign content elements to the application tree. This allows for a more clear and real view of the results.
• Support for widget-based and Mashup sites: The new Content-Based configuration (see previous item) view lets you define the structure of widget-based and Mashup sites and display their structure logically.
• WebSphere Portal support: Dedicated template for WebSphere Portal applications incorporating a WebSphere Portal Test Policy and other configurations designed to increase performance and accuracy. The same capability can be adjusted for other Java Portlet based web applications
• Improved Web services support: The new GSC utility replaces "Web Services Explorer" (a WSDL analyzer that generates SOAP traffic) to provide improved Web Services scanning, including support for MIME attachments, WS encryption and WS signatures. This means you can now test SOAP Web Services that make use of WS-Security standards.
• IPv6 Support: no need to explain
• CVSS-based Severity Reporting & Configuration: AppScan is now capable of reporting vulnerability severity using CVSS. In addition, users can modify CVSS settings as they wish, in order to create more accurate reports

These are just some of the major improvements and new features in AppScan Standard Edition v7.8

You can download a trial version of AppScan here.

BTW - for those of you who haven't been following our recent product announcements, we also recently shipped AppScan Developer Edition, which includes Static Analysis of JAVA (more languages to follow) applications, in conjunction with Dynamic (Blackbox) and Runtime Analysis. This composite type of analysis, enables developers to get a full view of the vulnerabilities, both from the web front end point of view, as well as at the source code level, in a correlated manner.

Automated Crawling & Security Testing of Flash/Flex Web Applications

Ronen Bachar, from our own IBM Rational AppScan team, gave a presentation on the (*hot*) subject of automated Flash/Flex application security testing, at the recent OWASP IL conference that was held last week (I have a strange feeling of a Deja-Vu).

The presentation gives a high level overview of Flash, Flex and the AMF protocol, and dives into some gory details (although some gore is missing from the online presentation) regarding the challenges and possible approaches for performing automated crawling and security testing of web applications that were built using these technologies.

 

And while we're on the subject of Adobe Flash & Flex web applications - if you happen to be in New York next week for the OWASP AppSec conference, be sure to attend another presentation by our team (Adi Sharabani & Ayal Yogev), on the subject of Flash Parameter Injection.

Blackbox vs. Whitebox - OWASP IL Presentation

Adi Sharabani and Yinnon Haviv, both from IBM Rational Application Security, gave a very interesting presentation on the subject of Blackbox and Whitebox application security scanners, at the recent OWASP IL conference that took place last week.

Since our team is working on both technologies, we believe that we can provide an unbiased and thorough overview of the difference between both scanning approaches, and also to point out how they complement each other.

You can view the presentation here, or download it from the OWASP IL conference site (will soon be uploaded).

We will soon upload another OWASP IL presentation given by our own Ronen Bachar, which talks about the challenges and solutions for automated Flash/Flex application security testing, so stay tuned!

My Wish for Open Source Web Application Security Tools

If web application scanning tools are the power tools used for broad application assessment, then the more sophisticated penetration tester will extend and refine the results through the usage of finely tuned scalpels.  Myself?  I've always favored using Netcat, Paros and human intelligence.  This is not to say that there are not many other powerful tools available, but these happen to be my scalpels of choice.

Whenever I hear people saying that they wish there was an open source web application scanning tool available, similar to a Metasploit type tool but for the application, I'm genuinely puzzled.  I wish for something even more basic - a solid, mature open source framework from which to perform web application assessments.  I want a framework from which I can begin with an architectural risk analysis, and move forward, collecting and trending SDL artifacts - through to a platform from which I can proxy, build, fuzz and report on my assessment.  Am I missing something?

Man Vs. Machine

Or: how to avoid the Assembly Line Syndrome

Recently, I’ve heard several security experts talk about the efficiency of automated web application scanners. Specifically, they raise claims that automated scanners are only good for:

• "Low Hanging Fruits" vulnerabilities
  
• "Technical vulnerabilities"
  

They all say that automated scanners cannot handle the "logical vulnerabilities". I thought it might be a good time/place to explain the difference between the types of vulnerabilities, and to explain why I think that every healthy security review of an application, should always contain both automated and manual assessments.