It’s easy to deconstruct a solution, syllogism, etc., and you clearly delight in that. That doesn’t change the fact that I’m one of the only people on the planet with real, hard numbers and measurements. Sure, those measurements are from “whitehat customers” but it’s the best current sampling on the planet. (debateable, of course)

Anyway, I’m working hard to give people useful tools and solutions to solve their daily problems. I think you are completely out of touch with business realities. I’ve heard your kind of rants before from people who lack ecommerce business backgrounds.

So this will be the last comment I see myself posting on your blog. We mostly agree on what the problem is, however, at this point I am going to agree that we disagree, fundamentally, about how to approach the problem. I don’t see that changing unless you give me something pragmatic I can measure.

You of all people should praise the fact that WhiteHat gives people real, hard numbers they can use to learn, attack, criticize us, whatever. No one else is taking that step.

I’ve written courseware and I’ve trained developers and I’ve implemented SDLC programs and all that bullshit for more years that most of the pundits talking about this crap today. Touting that as the end game is a bunch of academic nonsense. You could of course prove me wrong with some relevant, modern measurements. I want both knowledge-based & skills-based measurements, and qualitative and quantitative analysis of defect reduction and quality improvements.

And then that turns into its own circular game. Go read The Mythical Man-month, or some of Spolky’s essays on quality improvements. meh.

I would be talking about how great WAF’s were if they worked as advertised.

the example that you used, the massive SQL injection attack is a perfect one, this attack was easily stopped by using a WAF, a WAF can protect you from a simple SQL injection attack or a blind SQL injection attack, it doesn’t really care

Incorrect — the SQL injection attack (many different types of tools and threats were used) infected hundreds of thousands of web applications, which in turn infected thousands of Windows machines, turning them into a botnet. Although I expect that at least one or two F5 customers might have been protected against the second or third wave of these attacks, if they had a web application security expert configure their WAF properly.

However, I think those organizations would have been better saved by coding correctly in the first place, or even fixing the code when it popped up as an issue. They may have spent less time and resources — and they wouldn’t have to spend $110K for a WAF pair to do so.

however for an enterprise, real life poses a few challenges

Can you explain what those challenges are? I see WAF installation (configuration, verification, troubleshooting), operations, maintenance, etc as much more challenging than fixing the code.

WAF provide a way to raise dramatically the overall security of a website and provide the security officer a way to enforce it […]

What about the risks to the web application that the WAF does not protect against? What about the vulnerabilities that it cannot enforce? Millions of web applications are vulnerable to XSS, including every single one that sits behind a blocking WAF. Many XSS vectors go past the WAF detection modes as well, removing the need for a security officer to be woken up during the night. I don’t see this as beneficial.

Do you know how many enterprises are not developing their web applications?

I think that I have a pretty good idea having worked with many enterprises for a very long period of time, specifically regarding web applications and their vulnerabilities. When companies acquire applications (or software such as third-party libraries, components, frameworks, et al), they have to consider the risks. Managing those risks can be done in numerous ways.

how will they secure them?

“Test and verify before you buy / acquire”. For example by using third-party full-knowledge vulnerability assessment services, audits, evaluations, rating systems, assurance levels, et al.

many times, if they contact the vendor who coded them (hopefully the vendor is still in business) with a vulnerability report they discovered they get an answer, “yeah, we will fix that in the next release”, “oh”, “BTW, the next release is within 8 months”

It doesn’t sound like you’ve done this before. Have you worked as a vulnerability assessor? As a risk assessor? Certainly there are problems such as you mention (people do say those things), but there are ways to overcome them.

What about the web server itself? should they also fix the code of the web server?

I’m not talking about web servers. I’m talking about web applications. There is a very big distinction between the two. If you work for a WAF vendor, I suggest that you read up on the differences. You seem to mix the terminology as if they are one in the same.

Patches, yes, patches are very important and everybody should deploy them, however, it takes time until vendor supplies patches to known vulnerabilities and it also takes time until an organization can schedule a maintenance window to apply these patches (oh and of course, test them before that…)

Patches are what you install on the Operating Systems for the desktops and laptops that access web applications over a network. Patches prevent “client-side exploits”, and they often do not need to be installed right away if there are other controls in place to support a proper vulnerability management program. However, this again — has nothing to do with web applications. You seem to bounce around a lot, off-topic from the original conversation. I do understand why client-side exploits are relevant to this conversation about the SQL injection attacks (the botnet malware), but I fail to see how this has anything to do with WAF’s.

Logging and forensics, the logging which a web server is doing is very limited

Ok, so this is where tcpdump or socket or system-call tracing/logging comes in useful (or maybe a different web server). I’ve been using these types of tools for almost 15 years. Why do I need a WAF to do this?

in the real life of enterprises, WAFs are becoming a major building block, like firewalls 15 years ago

If this is true, I find it sad. Enterprises already have enough problems with respect to security. Adding a WAF into the operational picture must be a nightmare and a curse for these organizations. I don’t really relate the firewall and WAF technologies. First of all, firewalls are not as important as they once were because attacks come at the client-side through crimeware or other malware threats. Secondly, firewalls actively blocked network traffic. If a WAF blocked traffic, then I wouldn’t be able to go to the websites I like to go to. What do WAF’s block besides legitimate traffic and WAF vendor test cases? Do they block actual attacks? What kind of attacks do they fail to block? My guess is that WAF’s fail to block many, many, many kinds of web application vulnerabilities. We have a blog post about this called, Web application firewalls: a change of heart that you should definitely check out. It talks about a variety of even stupid check list stuff that WAF’s don’t block.

The real question is if WAF can significantly contribute to website security, the example that you used, the massive SQL injection attack is a perfect one, this attack was easily stopped by using a WAF, a WAF can protect you from a simple SQL injection attack or a blind SQL injection attack, it doesn’t really care.
Your assumption that the only solution to web application security is code fixes doesn’t stand in real life, you are missing the business problem aspect of this approach, you are also missing the challenges that enterprises have with this approach. Yes, if you have a blog and small web application that you maintain you could probably fix what you know about in the code, however for an enterprise, real life poses a few challenges.

1. The security officer is responsible for website security, however, he really doesn’t have any tools to enforce security, yes, he is going to the dev team and teaches them about security, yes, he is scanning the website and finds vulnerabilities, but how is making sure that the vulnerabilities are fixed? he doesn’t manage the dev team, the dev team will not get fired if the website was hacked, right? dev will always look at vulnerabilities like “bugs” and who doesn’t have bugs? on the other hand, who will get fired if the website was hacked? Who can’t sleep at night knowing that he has open 10 SQL injections vulnerabilities across his website that he needs to secure? WAF provide a way to raise dramatically the overall security of a website and provide the security officer a way to enforce it and sleep well during the night…
2. Do you know how many enterprises are not developing their web applications? (clue - every enterprise is using at least one web application that isn’t home grown… well, in real life, much more than one..) how will they secure them? many times, if they contact the vendor who coded them (hopefully the vendor is still in business) with a vulnerability report they discovered they get an answer, “yeah, we will fix that in the next release”, “oh”, “BTW, the next release is within 8 months”
3. What about the web server itself? should they also fix the code of the web server?
4. Patches, yes, patches are very important and everybody should deploy them, however, it takes time until vendor supplies patches to known vulnerabilities and it also takes time until an organization can schedule a maintenance window to apply these patches (oh and of course, test them before that…)
5. Logging and forensics, the logging which a web server is doing is very limited, what about logging of headers, POST data? you realize the benefit of such logging.

So thank you for raising the awareness for web application security issues, in the real life of enterprises, WAFs are becoming a major building block, like firewalls 15 years ago. Don’t get me wrong, I am not against fixing code, I just wanted to make sure your readers understand the challenges of enterprises and why they choose WAFs . By the way, I have to admit here that the company I work for, F5, is selling a WAF.

Your conclusions contradict all the real world measurements I can demonstrate. The SQL Injection bot is a perfect example of where blackbox scanning + WAF integration solves for businesses issues cheaply.

I would never trust measurements solely from a vendor. If MITRE comes out and says that WhiteHatSec is the gold standard of modern metrics then hell will probably also freeze over that same day.

A perfect example? How? I just showed how it was a horrible example!

Out of the 640-some websites I measured during these attacks, only ONE input vulnerable to SQL Injection (out of tens of thousands attacked by this worm) was missed using a blackbox scanner. The Sentinel platform to be specific.

You can’t even start understand the problem if you think that it only has to do with you and your customers. 640 websites? Please. Yes, out of tens of thousands, that’s exactly the point.

The missed input was a “Blind SQL Injection” as many of these are. During the period of this worm we implemented a number of new syntax arguments that now successfully find the one missed issue

You can’t find the issue with a scanning tool or even a manual pen-test. The issue is lack of parameterized queries.

We also can create blocking rules for all of the issues we discover.

Seriously, Arian, please tell me that you don’t believe that! How could a WAF possible stop a business logic attack? How could a WAF stop second-order attacks? I think WAF’s don’t even stop the most common attacks.

A WAF prevents what the WAF vendors (and WhiteHatSec) want you to believe that they will prevent. Firewall vendors don’t go out and research how their firewall doesn’t stop certain kinds of packets under certain kinds of scenarios. They don’t explain the client-side attack vector (and the extremely high likelihood of it) to their potential customers.

WAF’s can block `known’ vulnerabilities, but in the case of “what a web application security scanner finds” — the results are not then “known vulnerabilities”, but instead should be thought of as “known software weaknesses”. Once a software weakness is exposed, this means that there are still many vulnerabilities yet to be uncovered.

Look, if web application security scanners were like regular black-box functional quality tools — they would see around 30 percent of the total bugs. Instead, they see 7, 8, maybe 10 percent (we must still have a lot of work ahead of us!). The only way quality testers were able to push their black-box tests above the 30 percent mark was to utilize pairwise testing as well as techniques that can manage keeping track of state-transitions. State-transition technology is not available in commercial web application security scanners, neither is pairwise testing. I’ve heard an argument that it’s impossible to even properly utilize pairwise testing for security property testing purposes. Even with these additional testing techniques, black-box functional testing with automated quality tools only rises to 70, maybe 80 percent coverage at maximum.

These claims on Jeremiah Grossman’s blog about 1 percent false positives in Sentinel are also wishful thinking. We all know that the false positive rate is highly dependent on the application in question. Let’s see how Sentinel fairs if I setup some apache configs (in an already huge and highly complex application) to respond over HTTP in varied ways (such as 200 OK to everything). There’s no rule of thumb for false positives and you can’t take in averages. Everything has to be customized.

In fact, this brings up a great point. WAF and WASS technology requires a seriously large investment in customization by an expert. We don’t have enough of these experts to go around. So do we start with WAF/WASS training, or do we instead just work with developers and train them in the first place — since they are the ones empowered enough to do something about a root-cause fix.

There are plenty of websites that use WAF that have been hit with SQLi or XSS. Some of these websites employ a large department of WAF and WASS experts. The adversary who found the SQLi or XSS typically is not an expert — often they are kids with 1 or 2 years programming experience, and with no other security experience — nobody is paying them — they are unskilled, unknowledgeable about web applications, etc. It is easy to find SQLi and XSS if you understand the code. It is easy to fix SQLi and XSS if you understand the code.

It’s difficult to use a WASS to find weaknesses or properly attack an application unless you customize it and you’re an expert. It’s even more difficult to deploy a WAF in prevention-mode, in order to prevent known vulnerabilities — let alone the root-cause issues which are systemic to a certain problem in the code. Why would we waste time on this approach? To keep WhiteHatSec and WAF vendors in business? Any other reason?

I could go back and get you the ratio of missed issues to inputs if you like. It’s probably 1 per hundreds of thousands of inputs; 1 per millions if I include known non-vulnerable code (to the bot)

Did you check for second-order injections? I guess not since Sentinel doesn’t even support them.

Even at 1 per 640 sites….that’s pretty good. That’s 639 winners in this lottery.

Wow, aspiring customers win a $110k WAF price-tag (plus $12K/year for support) combined with a $25K/year price-tag for a SaaS WASS solution! But, wait, they still are weak against the OWASP Top 10?! And what’s more is that adversaries are already preparing themselves for their next avenues of attack. Guess what? It’s still at the web application layer!

If someone else can actually demonstrate, and not pontificate, a better working solution…I’d sure love to see it demonstrated.

You are really opposed to the idea of fixing the code, aren’t you? It cuts into the big numbers on the income sheet and drops holes and bombs all over that business plan? Only if it wasn’t so difficult to get bought-out by a WAF vendor (since you couldn’t sell to an HP or IBM) — why they have to be having these kinds of issues at the same time??

I was anti-WAF too at one point, because they are surrounded my myth and marketing FUD.

I am still anti-WAF. Do you really think that’s going to change anytime soon? What is the problem you are trying to solve by posting this comment on our TS/SCI Security blog?

I enjoy the discussion; I love the quick wits — but do you really think you’re going to be able to convince me that concentrating all effect into fixing code is the wrong thing to do?

Hear me out for a second. Do you really think that they are ZERO quick wins that can be accomplished by involving the development groups in any given organization?

What about with specific regard to regression testing? I know that WhiteHatSec has customers that have been in regression testing for over 2 years. If you look at the WhiteHatSec numbers for Time To Fix, do you really think this applies to anyone EXCEPT WhiteHatSec customers?

To me, these numbers show what a horrible job that WhiteHatSec is doing. Maybe the problem is you?

I can think of dozens of ways to improve regression testing with regards to any sized development department — that would instantly result in a better understanding of reoccurring bugs. Within a few days, organizations get to see instant results if Continuous Integration (or some form of it) can gain ground and mindshare. With test-first development, bugs start to magically disappear, while at the same time — time in regression goes from months to hours.

Instead of forcing two things that go bad together into an abomination/monster — WAF and WASS, I’m trying to come up with ways of making other wins work well together, such as CI (continuous integration) and TDD (test-driven development).

That’s why we took all the mystery & magic out of them by creating targeted attack vector blocking for known vulns. It is very simple, no magic, and like most simple solutions — it works well.

Let me see how it works under the hood. It sounds simple on paper, but anyone who has worked with a WASS or WAF knows that it’s not. Even if WhiteHatSec was the best WASS solution (which it’s not), and even if WHS did hire the best talent then there would still be problems.

Arian, you are the best talent out there with regards to web application security. Jeremiah, Anurag, and at least 4 other people that I’ve spoken with at your company are also of extremely high caliber. But that’s 7 people, only one of which is really anything outstanding. 7 people and 640 websites is bad math.

The way that you focus your expertise and skills is just misguided. You should be working with developers, doing training, or getting the right ideas out there into the community by giving back.

If you’re not a part of the solution, you’re part of the problem.

Out of the 640-some websites I measured during these attacks, only ONE input vulnerable to SQL Injection (out of tens of thousands attacked by this worm) was missed using a blackbox scanner. The Sentinel platform to be specific.

The missed input was a “Blind SQL Injection” as many of these are. During the period of this worm we implemented a number of new syntax arguments that now successfully find the one missed issue.

We also can create blocking rules for all of the issues we discover.

I could go back and get you the ratio of missed issues to inputs if you like. It’s probably 1 per hundreds of thousands of inputs; 1 per millions if I include known non-vulnerable code (to the bot).

Even at 1 per 640 sites….that’s pretty good.

That’s 639 winners in this lottery.

If someone else can actually demonstrate, and not pontificate, a better working solution…I’d sure love to see it demonstrated.

I was anti-WAF too at one point, because they are surrounded my myth and marketing FUD.

That’s why we took all the mystery & magic out of them by creating targeted attack vector blocking for known vulns. It is very simple, no magic, and like most simple solutions — it works well.