Web application firewalls: A slight change of heart
We’ve been beating the drum for some time now, expressing our opinions of web application firewalls (WAFs). You might have sided with us on this issue, are against us, or are just tired from it all by now. This post is about to change all that, and show that we are not 100% anti-WAF, and that there are some useful applications for them.
Why WAFs do not work
In a post on why most WAFs do not block, Jeremiah Grossman quoted Dan Geer:
When you know nothing, permit-all is the only option. When you know something, default-permit is what you can and should do. When you know everything, default-deny becomes possible, and only then.
Jeremiah then stated that to implement a default-deny WAF (which would offer the most security, but carries with it the greatest business impact), you need to know everything about your app, at all times — even when it changes. How appealing is this, given the amount of resources you currently have? Who will be responsible for maintaining the WAF? These are all questions you should be asking yourself. Jeremiah then goes on to say that default-permit is necessary in web applications — going against everything we’ve learned in security over the past 40 years. Wait… what??
Some context that can attest to our reasoning
Over the last several weeks, I’ve been evaluating several web application firewalls. They all have their own list of cons and more cons. On my first day, having sat down at their consoles, I was a little overwhelmed by all the options present — application profiles, signatures, policies, etc. It all came to me as I worked through it and read the manuals, though frankly, I don’t see how anyone without a web application security background can keep up with it all. I fear these devices will be deployed and forgotten, relying solely on their ability to learn and self-adjust.
Let’s talk about the consoles used to monitor and maintain the WAF. One vendor had a fat app, which was a bit clunky, non-intuitive and had multiple usability issues. The number one issue that comes to mind is on the monitoring panel — to watch alerts in real-time, you need to set an automatic refresh rate which updates the entire display, which makes it impossible to analyze HTTP requests/responses during this time. If you’re scrolled down to a certain location of a request, and the console refreshes, you lose your position and are brought back up to the top. I don’t understand why the entire screen had to be updated, rather than a particular frame.
Another vendor used a webapp to manage itself, which was in my opinion much nicer and easier to use, albeit slower. When on the alert monitoring page, you had to manually click a button to refresh the alerts, and viewing requests/responses was a major pain. The application utilized AJAX on pages that could do without, but in areas that could benefit from them, they resorted to old web tactics.
In the course of my testing, I started by taking RSnake’s XSS cheatsheet and creating Selenium test cases for attacking our own vulnerable web application (See our talk, Path X from ShmooCon). For those unfamiliar with Selenium, it’s a browser driver that performs functional testing, though we have showed how it can be used for security testing. We didn’t use WebGoat (or any other vulnerable apps), reasoning that the vendors must have tested against those and know them inside out for just such occasions. Renaud Bidou had an excellent presentation on How to test an IPS [PPT] from CanSecWest ‘06 which I believe can be applied to testing WAFs for those interested. Suffice to say, the WAF’s did not detect ALL of the XSS from the cheatsheet that was thrown at it, which is pretty sad. I would have expected they at least get that right.
That brings us to second-order, persistent XSS and SQL injection attacks. When a web application strings together data from multiple sources, detection of such attacks can be very hard. The WAF cannot account for this logic, thus allowing an attacker to effectively bypass the WAF by staging his XSS/submitting multiple payloads to various sources. When the application then pieces the data together, an XSS (SQL injection, etc) condition exists. The problem with this? Your WAF never detected it, and you have no idea your site’s been attacked and is now hosting malicious scripts.
There are just some attacks a WAF will never detect. HTML / CSS injection through HTML / CSS is just one example. Go on over to http://google.com/search?q=cache%3Atssci-security.com — can you describe what is going on here?
Or how about CSRF? Insecure session management? What can a WAF do to protect against business logic flaws? We can go on and on, and yet vendors still claim protection against OWASP Top 10, which if you believe shows you know nothing about web application security.
How WAFs can help
So I lied, we haven’t changed our minds about WAFs. But wait! I’ll let you know what would change our minds at least a little, which would show that WAFs can have their purposes. Without this though, I can’t recommend any organization spend the money on such devices — especially if they need to meet compliance requirements where other options do exist.
The value of WAF Egress, revisited
What should a WAF do? Block attacks on the egress / outbound, while staying out of the inbound flow of traffic. I’m not talking about signature based blocking either. This is the tough part, because it’s almost impossible. One way I see it working though, is if the application keeps the content (HTML), presentation (CSS), and behavior (JavaScript) separated. The application should not serve any inline scripts, but instead serve script files that would alter the content on the client side. This would make, e.g. outbound XSS prevention possible because a WAF could then detect inline scripts in the content. None of the WAFs I evaluated could detect a client being exploited by a persistent XSS condition. This would also tell me how many users were affected by the XSS attack, which we haven’t seen any numbers on apart from the number of friends Samy had when he dropped his pants and took a dump all over our industry.
Jeremiah and I got a picture with him wearing “Samy is my hero” shirts. I haven’t laughed that hard in a long time! But to quote a sanitized version of what one guy said, “Samy knew nothing about webappsec and one day he walked in, dropped his pants and took a huge dump on our industry and then left again. And we just looked around at one another and said, ‘What just happened?’”
Another way to get this right is to apply the work done by Matias Madou, Edward Lee, Jacob West and Brian Chess of Fortify in a paper titled: Watch What You Write: Preventing Cross-Site Scripting by Observing Program Output [PDF]. They go on to talk about capturing normal behavior of an application during functional testing, and then attacking the application as if in a hostile environment, where it is then monitored to ensure it does not deviate from normal behavior. Basically, it’s all about monitoring your application output in areas that are known to be dynamic.
In-depth, the Foritfy work is using dynamic taint propagation, by which “taint propagation” or “taint tracking” is similarly done with static analysis in order to trace misused input data from source to sink. This is also a corollary to the work that Fortify has presented on before with regards to Countering the faults of web scanners through bytecode injection [PDF]. While web application security scanners only demonstrate 20-29 percent of the overall security picture because of surface and code coverage for the inputs of the application under test, dynamic taint tracking goes a long way to providing more coverage for these kinds of tests because it’s done as white-box dynamic analysis instead of functional black-box runtime testing.
The value of XHTML
My fellow blogger, Andre Gironda, helped out with the praise section for the book, “Refactoring HTML: Improving the Design of Existing Web Applications”, by Elliotte Rusty Harold. It’s hard to disagree with the notion that XHTML can help with both quality and security issues, as well as make applications and content easier to refactor and work with.
When you’re recoding thousands or millions of lines of code, wouldn’t well-formedness and validity be the primary requirements for working with such large volumes of code? If anything, well-formedness and content validity make the chores much easier to deal with. Rusty has this to say in his book:
[…] there are two things [authors for the Web] are very likely to write: JavaScript and stylesheets. By number, these are by far the most common kinds of programs that read web pages. Every JavaScript program embedded in a web page itself reads the web page. Every CSS stylesheet (though perhaps not a program in the traditional sense of the word) also reads the web page. JavaScript and CSS are much easier to write and debug when the pages they operate on are XHTML rather than HTML. In fact, the extra cost of making a page valid XHTML is more than paid back by the time you save debugging your JavaScript and CSS.
Since web application firewalls today cannot convert HTML on the outbound to XHTML, this is certainly a job for the content writers (sometimes, but often not the developers) to deal with. In the Refactoring HTML book, Rusty also talks about the tools necessary to develop content on the web:
Many HTML editors have built-in support for validating pages. For example, in BBEdit you can just go to the Markup menu and select Check/Document Syntax to validate the page you’re editing. In Dreamweaver, you can use the context menu that offers a Validate Current Document item. (Just make sure the validator settings indicate XHTML rather than HTML.) In essence, these tools just run the document through a parser such as xmllint to see whether it’s error-free.
If you’re using Firefox, you should install Chris Pederick’s Web Developer — https://addons.mozilla.org/en-US/firefox/addon/60 — plug-in. Once you’ve done that, you can validate any page by going to Tools/Web Developer/Tools/Validate HTML. This loads the current page in the W3C validator. The plug-in also provides a lot of other useful options in Firefox.
Whatever tool or technique you use to find the markup mistakes, validating is the first step to refactoring into XHTML. Once you see what the problems are, you’re halfway to fixing them.
Speaking of properly validated and easy to read/use content, what irked me throughout my evaluation most was documentation. Vendors: do not bundle a ton of HTML files together and call it a manual. If you’re looking to do that, please use DocBook if you’re not going to make a PDF available. Better yet, give us a hard copy.
Good post. Some points:
1)
>Since web application firewalls today cannot
>convert HTML on the outbound to XHTML, this is
>certainly a job for the content writers
>(sometimes, but often not the developers)
WAFs could do this pretty easily using existing open source libraries. O(n), too. Pure XHTML output could allow a lot of security to be introduced, including WS-* type of lunacy if people want to go that direction (please don’t).
2)
>It all came to me as I worked through it and
>read the manuals, though frankly, I don’t see
>how anyone without a web application security
>background can keep up with it all. I fear these
>devices will be deployed and forgotten, relying
>solely on their ability to learn and self-adjust.
This is a similar issue that the scanners face. Their original purpose was to make security testing “easier”, but they end up costing the same or more than manual analysis with all the overhead of setting up a meaningful test and chasing down false positives, while only catching 20%-30% of the issues.
In the case of WAFs, the hidden-cost-result is a bit different. The complexity of getting useful security out of a WAF prevents most organizations from getting real value out of it because it will not be a webappsec expert maintaining it (unless the market has produced a million of these in the last year). especially for new apps.
The lack of documentation you cite only makes things worse. Thankfully, this is something they can do pretty quickly.
3) I’m gonna have to officially claim the Samy-taking-a-dump-on-our-industry quote if people are going to cite it everywhere. =)
4) I’d love to see more details on everything here, including what of their attack coverage is missing, why they’re missing it (is there a business reason?), etc.
5) Review the scanners next! Arian’s work is getting antiquated now.
@ arshan:
WAFs could do this pretty easily using existing open source libraries
TagSoup, Tidy, NekoHTML, and others are great — but there is always the problem of having to “spot check” and fix some problems manually.
Some XHTML conversion tasks are easier than others. Tidy or recode will turn text into UTF-8 in a snap, but Tidy/TagSoup fail to reliably fix quotation marks inside attribute values. While TagSoup won’t add a strict DOCTYPE, Tidy does a great job.
Really, though, this shouldn’t be done with an appliance. If 30 percent or more web designers utilize Dreamweaver (and many more use similar tools that can output valid XHTML), then the problem exists between chair, keyboard, and manager.
I’d love to see more details on everything here, including what of their attack coverage is missing, why they’re missing it
Well, Marcin did cover some of these things. It would be nice of the vendors to provide a list of what WAF’s cannot do for us. They should be the ones doing this work. Why should end-users and consultants have to do it for them?
Review the scanners next! Arian’s work is getting antiquated now
I have come to the realization that web application security scanners are awareness tools (although I also think that security review tools based on static analysis techniques are also mostly awareness tools).
In this case, scanners need to have the ability to raise awareness. There are lots of web applications out there, many different sites, many virtual servers, etc. Some of the most important capabilities of a scanner would therefore be surface coverage and speed. Accuracy isn’t quite as important unless getting source code is really a no-no, in which case you should just drop the customer and tell them to fix it on their own. In my own comparisons, HP WebInspect topped the fastest, most reliable scanner with good surface coverage and reports; Cenzic Hailstorm as the most accurate — with the least amount of errors over a board range of applications. For those who like vulnerability scanners, I tend to think of McAfee Foundscan and WebInspect as the most useful for everyone; while nCircle IP360 and Hailstorm as the tools for the real experts — when the kid-gloves come off.
I also have to agree with Jeff Williams in that scanners don’t need to be run if you know that there is nothing in the SDLC or known by the developers that will prevent software weaknesses. The best way to handle this sort of thing is to do some interviews with the developers, possibly a code walkthrough, talk about the design and language choice, or even get some bits of code to look at.
Personally, I think that if you can get the entire source codebase (i.e. one that builds), then using Fortify SCA, Ounce, Klocwork K7, or GrammaTech CodeSonar is paramount to running a web application security scanner. If not, then Checkmarx CxSuite or GrammaTech CodeSurfer (or possibly the lesser known parts of Klocwork K7, such as the architectural analysis mode) could instead be used on parts of the unbuildable source. Even Pixy, RATS, PHP-SAT, SWAAT, and Orizon could be used as open-source alternatives in a pinch. Out of all of these tools, myself and others usually prefer Ounce and CxSuite (although Fortify SCA does have some crazy cool language support if you aren’t looking at just Java or C#).
Better than using an automated security review tool would be manual code review. I have to say that Guidance Explorer (for .NET) and Jupiter (for JEE) are above par as open-source tools for manual code review purposes. I have not used GrammaTech CodeSurfer, but the screenshots and feature list look great, as well.
While the assurance that comes from manual code review is probably the highest, it also takes the longest. Automated security review tools that do static analysis (either the RATS “grep” way, or the Foritfy SCA `AST parser’ way) can be useful, but they also require an expert in both security and development. Some may argue that in many cases (and I tend to argue this), that bytecode techniques (Fortify SCA and Ounce employ some of these as well) are much more useful. Take the FindBugs (Java), and FxCop (.NET) tools as an example. The security checks from these tools are quite good. Veracode even adopted this sort of model for their own SecurityReview service.
However, this is all just to raise awareness, or as a “one-cycle” answer to the security problem. These problems don’t solve the root-cause of the issues, or prevent developers from making mistakes long-term. I really feel that the only power that developers have (especially quick wins, which manual code review is not) is to utilize integration unit testing both inside their IDE as well as on a build server. The integration unit tests should test for security properties. In order for this to work for Eclipse, you’ll need Apache Cactus, PicoContainer, Google Guice, or similar. For .NET, there are packages from Microsoft such as Unity in ASP.NET 3.5 (or Pex) — as well as third-party support from tools such as MonoRail (for ASP.NET 2.0) and Ninject (my current favorite).
What I’d really like to see is a convergence of technology for WAF, WASS, and SCA. I think that dynamic taint propagation is certainly useful for WAF (as seen in CORE GRASP) — but why not integration with the language itself? Can PHP add taint tracking like Perl and Ruby have? Of course it can! Can Java? Yes, of course. Web application security scanners should be able to add dynamic taint propagation as well. SCA tools should get better at automated DFA, with less errors across a larger variety of languages, frameworks, etc.
There’s lots of areas for improvement in all of these tools, but really none are as good as building a custom test harness into the developer IDE and build server. This missing piece is something that isn’t in the trade magazines, not talked about at SANS Application Security Summits, and never seen or heard of nearly anywhere in the blog world. Why is it kept so hush-hush? If the answer has to do with security vendors having to remove their FUD, change their marketing, and begin to drop focus on primary products — then this is a sin of our industry. We are liars — crack dealers — selling temporary or not fully built/realized solutions.
For WAF and WASS vendors, this is even more of a sin than for security review tool vendors. However, it needs to stop — we need to look at where our technology is and admit that it’s hurting us more than helping us. This also goes for training, but that’s an entirely different conversation, although related to the distinctions I made about what developers must do (test) vs. what is nice for them to be doing (review).
I’ve talked to Marcin - nothing would be better for helping them do the things they’re supposed to do than release a number of vectors that trivially bypass their detection (whether or not they’re in IPS or IDS mode). But I know you’re not like that.
We’ve been recommending integrating SCA into various build processes for a while - it’s a no-brainer. If money was no issue it would probably be standard.
RE: egress. One of the most useful things an egress filter like a WAF can do is stop an XSS worm or SEO attack very early, assuming some active outside detection is going on, or some application triggers are in place to detect fishy behavior. Depending on the customer, that alone may be reason enough to have one.
One thing you have to realize, before you give yourself an aneurysm, is that the marketing people won’t change their tune. They might tone it down a little bit if you put enough work into it, but you’re just putting up sandbags against the tsunami.
This isn’t a problem within webappsec - it’s the way the world works. Evangelizing is good, especially if we have big bullhorns, but really the market has to figure out for itself the niche uses for scanners and WAFs and then the market will correct itself.
@ arshan:
We’ve been recommending integrating SCA into various build processes for a while - it’s a no-brainer. If money was no issue it would probably be standard
What does money have to do with it?
Money for the people and time invested? Or are you talking about the money involved in the cost of the tools?
Just look at this list of open-source solutions that apply for Java/JEE, C#/VB ASP.NET, and PHP respectively: PMD, FindBugs, Orizon, & SWAAT (with LuntBuild) — StyleCop, FxCop, XSSDetect, & SWAAT (with CruiseControl .NET) — PHP-SAT, Pixy, RATS, & SWAAT (with Phing). Add EMMA, PartCover (or NCover for ASP.NET 2.0), and PHPUnit3 with XDebug for code coverage reports. Again, all open-source.
Note that this is just what is available for SCA. For unit testing and integration unit testing — there is quite a lot more! In fact, there aren’t many commercial solutions for integration unit testing and dependency injection frameworks. Now you have me thinking if the lack of commercial product support is the reason why companies have failed to adopt these practices.
Or maybe it’s because the potential customers of those solutions are spending too much money and time on their WAF’s.
> What does money have to do with it?
The question is more: what doesn’t money have to do with it? Time, money, and opportunity cost involved with getting a corporation to approve the use of an open source tool, training to use it, and then living with severe gaps in coverage?
The commercial SCA is the same question, except the gaps are smaller and the money cost is higher.
@ arshan:
Well if commercial SCA is really too expensive, you could always hire out a third-party consulting company who already has a commercial SCA to do code review checks for you.
Crappy Google Adwords only shows AspectSecurity and Veracode come up for keyword searches, but I guess that’s better than nothing.
It’s interesting to know that WhiteHatSec has similar Google Adwords out that says “June 30 PCI Deadline”. That’s kind of cute.
In all seriousness, I don’t see how money isn’t an issue with everything — with WAF — with WASS — with saying the word “security” in the wrong context and causing a hallway debate. But that wasn’t my question.
I’ll recap, because I kind of lost you. I just commented because in your post you were wondering why people weren’t clamoring integrated SCA.
I was letting you know that we were, but money is the main pushback I’ve heard from people, which is typical as security is usually underbudgeted.
Then you started ranting. =]
XOXO,
Arshan