« There's a reason they call it the OWASP Top 10 | Main | What The Fuzz Are You Talking About?! »

May 28, 2007

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d835130c5153ef00d8357bb1e669e2

Listed below are links to weblogs that reference Man Vs. Machine:

Comments

Arian Evans

Let's break down the logical examples more clearly. They fall into a couple of categories:

1. What scanners can't do today, but *should* do tomorrow:

Examples of this include authentication issues, or obvious authorization issues (binary decisions about functionality access)

2. What scanners can't do today, and will have a hard time ever doing accurately:

Examples of this are subtle authorization issues, like SQL LIKE queries sitting on top of a lose authorization model. Without a threat model/mis-use case described in the scanner (which NO ONE on the planet does, outside of (some) professional assessment shops like I work for)...without either human eyeball context ("Oh, that's Bad!") or a pre-defined scanner rule or goal ('Rob can't query Sally's data) these will never be found.

3. Things scanners will never find:

Examples are like weak secrets used in password reset flow that includes email or out-of-band communications.

---

You are right: the emphasis on Getting The Job Done should be Man Plus Machine. I definitely agree with you there, and I bet Jeremiah does too.

However, there's still a bit of crappy hype and automation fantasy out there, and we need to objectively and clearly qualify and measure what is what, today, goals for tomorrow, and what bar we hit, to make this "art" the science it should be.

Nice blog, btw//

-ae

AppSecInsider

Arian,

Your examples are a bit high level, and it is hard to understand what specific vulnerabilities you are referring to.

Let me ask you a question -
Do you think that tampering with a user ID value in a cookie (e.g. decrement the value), and accessing someone else's account details, can be automated or not?

From what I've heard and read, most people will say that this is too hard to automate (i.e. "how would a piece of software figure out that it switched context", right?)

I believe that such a scenario can actually be automated, given a good enough algorithm.

In general, I think that some of the tasks that you and Jeremiah refer to as hard or impossible, can actually be automated, and I believe that people should use the term "Halting Problem" in this context, a bit more carefully.

Arian

Example #1 is high level. The others are very specific, which is why I gave specific examples:

Your cookie tampering falls sometimes into #1, and other times into #2. It depends upon the cookie and the site, entirely. (unless you want to generate a ton of false positives)

1. Binary authorization decisions: should auth state A have access to /admin/manageusers.asp?

Yes|No. Scanners could and should do this. Today they do it between poorly, and not-at-all. A few try, but I have yet to see it done well. Soon I suspect. We're pretty good at this sort of thing.

Is authentication state, or authorization access, different between cookie 1001 and 1002 in a meaningful manner?

That can be automated, I suppose. But that isn't the authorization problem today, most commonly seen, IMO (circa 2007).

2. Authorization decisions: cookie Arian and cookie Ory. Can I swap them inside a valid session and bypass authorization checks? Yes, this can be automated, if you describe the problem up front. No, scanners are never going to automate that in click-scan-mode or while using only one set of user-creds. If the scanner is using multi-user auth, in some cases, yes.

In the case of functionality that sits on top of a weak (or non-existent) authorization model, that can be abused within the boundaries of that functionality: I gave the example of SQL LIKE queries that can be manipulated to return other user's data. That exists out in the real world. We find it. Scanners do not, and will never be highly accurate at this.

Unless you have a pre-defined mis-use case. Which I don't think anyone is ever going to give scanners.

3. Never going to find category: Again, I was very specific: if I can break your authentication, by resetting passwords to your accounts through your password reset functionality, and that involves weak secrets used in the decision making:

1. Scanners are never going to be useful at identifying weak secrets that are non-integer.

2. If there is OoB (out of band) decisions made, like in the case of a password reset email, today's scanners will never collect that data set to analyze.

This is not about having a stance to defend. It's reality. I mean: we had one of our mutual competitors give a customer of mine the green light, full bill of health, all good, with their website. No Vulns, No Issues.

We we able to take over accounts through one vuln, and access GLBA-restricted data for all users through another weakness (this one involved binary image data, another thing scanners are almost useless at analyzing).

The site was fundamentally broken in multiple ways that no one is going to algorithmically solve any time soon, if at all.

I'll eat those words when proven wrong, no problem with humble pie here. I just don't see it happening.

I do agree that the Turing/Halting problem is thrown around way too much when discussing this.

I myself like to mention Rice's theorem as an even better example of the problem.

We need better stats and analysis of common vulns that "can be found", right? Then we could have this discussion in a more meaningful way, and mock up proof cases.

Right now it's sort of folks like me saying "you can't find that" and others say "yes I can" or "I don't know what you are talking about, do you mean: [insert trivial scanner automation problem]?".

I think it would be more productive to define and measure,

AppSecInsider

Hi Arian,

This is what I call a discussions :-)

1. Binary authorization decisions - your example is covered by AppScan's Privilege Escalation testing capabilities (check out my whitepaper on the subject at: https://www.watchfire.com/securearea/whitepapers.aspx?id=24 )

2. I don’t see a problem with automating a test that swaps cookies during session, to attempt to bypass authorization mechanisms. Although I do find this example a bit problematic – if I am switching session cookies, what did you expect would happen? (of course you'll switch sessions, that's what session cookies are all about, no?) Unless of course the session cookie, and the identity token (either cookies/parameter) are not the same one (in which case, like I’ve said – automation is possible to some extent)

3. Regarding the SQL LIKE queries, I would love to see some real world examples. That would definitely help to figure out why you think it is a tough one

4. Checking for weak password reset forms can be automated to some extent, for example, you can check if the form requires the old password, or you can check if access to that form requires you to be logged in or not, etc. – obviously not all cases are covered.

5. Regarding OOB responses – obviously, not all scenarios can be covered, but if the scanning software includes a useful extensions framework or an SDK (like AppScan does :-)), you can attempt to automate some of the peculiar scenarios, such as receiving one-time passwords for login via a mobile phone connection, or picking up information from emails, etc. – again, this is far from being perfect or complete, but it might increase the things you can do with the software.

Anyway, for the most part, I agree with you. There are things that scanners don’t do today, and they should be doing (we are working hard on improving all the time), there are things that will be hard to automate in the future, but I am sure we’ll get there eventually. And, there are things that require a human to figure them out – no argument here.

Thanks,
-Ory


Drexx Laggui

05Jun2007 (UTC +8)

Cool topic. "Faster, Better, Cheaper" is what I've adopted too.

lehitraot,
--Drexx

The comments to this entry are closed.

Follow us on Twitter

AppScan Free Trial


Try IBM Security AppScan software at no charge.

Become a Fan