What web application security really is
I wanted to do a post about “what web application security really is” because plenty of people out there don’t get it. They understand that “security attacks are moving from hosts to the Web”, but they have no idea what that means. To most people, web application security is the same thing as website security. I see people trying to approach web application security in the same way that they have tried host security in the past: penetrate (web application security scanner) and patch (web application firewall) — which won’t work.
Web application vulnerabilities are different from regular vulnerabilities in more ways than you would think
Web application vulnerabilities are not a one-time thing. In a similar way, buffer overflows are also not a one-time thing, however we’ve been lying to ourselves for 20 years about it. The nice thing about buffer overflows is that they have been historically found with random input testing (e.g. fuzz testing) or code review as “one-off scenarios”. A security researcher will find one, but that doesn’t mean it’s exploitable — better yet it doesn’t mean that the entire codebase is riddled with them (however, some repeat offenders obviously suffer from this problem).
With web application vulnerabilities, the more complex the app — the more likely an adversary can make his or her dreams into an exploit reality
In the case of web application vulnerabilities — one vulnerability means that there are often thousands of others — hence the claimed high rate of false positives in automated security review tools — with specific regard to SQL injection and XSS. What’s even more clever about web application vulnerabilities is that they often work together — they string together to form a bigger attack. A few little bugs equals one giant nasty bug. This also isn’t the situation with fat applications — a stack-based buffer overflow is usually one mean bug, but once it is patched properly the nightmare is [usually] over.
Web application profiling and Google Hacking read the developer’s minds like Jedi magic tricks
Another problem with web application vulnerabilities is Google. Yes, Google. Before Google’s code search, it was difficult to use Google to find a new buffer overflow. But such is not the case with web application vulnerabilities. All the blame isn’t on just Google’s search engine, but also MSN Live Search, Yahoo, and many others. It’s so easy to just find a login form and start playing with specific, exploratory characters. Buffer overflows require buffers of a certain length with enough room for shellcode to overwrite EIP (or execute via cute pointer tricks) in order to establish a reliable exploit. Web application vulnerabilities don’t worry about such restrictions. Once a web application is profiled (which could be as simple as an inurl:login.asp Google search), an intelligent adversary should already know numerous amounts of input techniques which will work against that specific type of application.
Web application vulnerabilities are like Legos for crimeware
Finally, adversaries aren’t moving away from client-side vulnerabilities. They’re just implementing the attacks differently. Web application vulnerabilities work well along with older host-based vulnerabilities. This isn’t only because web application vulnerabilities can get inside the firewall. They open up many different doors for attack. Using Inter-protocol exploitation, it is possible to send shellcode through an XSS/CSRF/XHR worm. But it’s also possible to send XSS worms through XSS worms or SQLi worms through XSS worms. Or XSS worms through SQLi worms. An adversary can put these types of attacks together however he or she wants.
Web app security today — The 2008 SQL Injection Attacks (January - present)
The 2008 SQL Injection Attacks have spawned a lot of talk and controversy about web application vulnerabilties. It appears that if web application vulnerabilities were not mainstream in crimeware — they now are.
The Microsoft Security Vulnerability Research and Defense blog had an excellent post regarding these SQL Injection Attacks. I think it dispells a lot of the myths about this attack, and it provide a lot of information on what to do about it. Of course, if you’re not used to the developer terminology (as an IT security professional or manager), it’s probably now as good as time as ever to read up, hire some experts, and get the word out.
There’s a lot of links, but the most important one is how to identify and fix the problem. What’s interesting is that a lot of the links talk about ASP.NET, but the attacks from Asprox’s fake “Microsoft Security Center Extension” SQL injection attack tool only seem to currently target Classic ASP.
If you have access to IIS logs, you can also run the SQLInjectionFinder.exe tool. Of course, if you run a web application security scanner, you may or may not find SQL injections in your Classic ASP web application — but this doesn’t necessarily mean that you are or aren’t hosting the Javascript malware. If you have a web application firewall, this doesn’t mean that your web application can’t be attacked via internal networks (assuming your web servers are listening on different interfaces) — and it certainly doesn’t mean that you aren’t hosting the Javascript malware.
- Web application security scanners cannot help in this scenario
- Web application firewalls also cannot help in this scenario
A better tool to test if you are hosting the Javascript malware would be to use a tool such as SpyBye.org. You can just set up your web browser to proxy through Spybye.org, or you can run it locally (e.g. to check your Intranet web applications) — and it can also integrate with ClamAV. TS/SCI Security (well, Marcin and I) discussed this strategy when we did the Network Security Podcast with Rich Mogull and Martin McKeay.
Many faces of the same SQL injection attack
Also mentioned in the Microsoft Security Vulnerability Research and Defense blog post on the SQL Injection Attack is that it’s not just one attack tool — it’s a bunch of tools and tactics. For those of you not familiar with Joe Stewart’s SecureWorks article on Danmec/Asprox, be sure to give it a full read. Additionally, check out the SANS article from Bojan Zdrnja on the 10,000 web site infection mystery solved. My favorite is a quote from eWeek’s article on Botnet Installs SQL Injection Tool:
Researchers are still investigating exactly what vulnerability on the Web sites is being exploited, Stewart said. The Web sites are English-language and their owners include law firms and midsize businesses.
A similar attack technique is currently being seen spreading game-password-stealing Trojans from China. Whether the tool is related or only the attack syntax is shared, it is clear that SQL injection attack activity is on the rise from multiple sources, Stewart wrote in his blog.
Why WAF’s are the wrong answer if you didn’t get it the first or second time
When I see WAF support from organizations like SANS supporting Jeremiah Grossman — as well as big companies like Imperva — I immediately question their reliabilty as sources for expertise. If the web applications serving malware are largely owned by law firms and midsize businesses, then these are likely outside of the scope of SOX, PCI-DSS, GLBA, HIPAA, and BITS Shared Assessments. Many of these organizations don’t even have the money or administrative staff to support WAF’s as an option.
What works in web application security: Take 2
What these companies need is an IT decision to remove, replace, or recode any web applications which clearly demonstrate an affinity for SQL injection vulnerabilities (especially ones written in Classic ASP). If replacement or recoding are used, then it would be a good idea to establish some software risk guidelines such as how web application software is acquired and tested before loaded by a web server. It’s not a matter of production or not — all applications, internal/Intranet/lab/test/etc — all of these applications need to be asset tagged and approved as tested. It’s now considered important enough to know what web applications your organization is running because of the risk involved by a SQL injection (or other web application vulnerability) “drive-by”.
Why we need action now
If we just let this malware sit dormant in web applications, we’re in for a lot of trouble. All that is required for adversaries is to flip a switch — and now they can deliver any new zero-day threat they desire. These could be websites you visit everyday. This could be web application code executed by your browser when you start it. It could even be your Intranet server, or a partner site you use. It could be your favorite online shopping site.
Don’t hesitate to make a decision. The next wave of these attacks may not work through an antiquated botnet or a standalone tool. They will probably target PHP MySQL applications, ASP.NET applications, both, or even more. There may not be simple tools such as SQLInjectionFinder.exe or SpyBye to help locate these vulnerabilities — and some could even stand the scrutiny of incident research for some time. How long did it take us to understand these SQL Injection attacks? Longer than 6 or 7 months?
If you think that implementing a WAF will save you (even in the short-term), please let us know why you believe this is the case. TS/SCI Security sees the WAF answer as FUD, lies, and/or short-sightedness. The best answer is to recode or replace while we still have the chance.
Your conclusions contradict all the real world measurements I can demonstrate. The SQL Injection bot is a perfect example of where blackbox scanning + WAF integration solves for businesses issues cheaply.
Out of the 640-some websites I measured during these attacks, only ONE input vulnerable to SQL Injection (out of tens of thousands attacked by this worm) was missed using a blackbox scanner. The Sentinel platform to be specific.
The missed input was a “Blind SQL Injection” as many of these are. During the period of this worm we implemented a number of new syntax arguments that now successfully find the one missed issue.
We also can create blocking rules for all of the issues we discover.
I could go back and get you the ratio of missed issues to inputs if you like. It’s probably 1 per hundreds of thousands of inputs; 1 per millions if I include known non-vulnerable code (to the bot).
Even at 1 per 640 sites….that’s pretty good.
That’s 639 winners in this lottery.
If someone else can actually demonstrate, and not pontificate, a better working solution…I’d sure love to see it demonstrated.
I was anti-WAF too at one point, because they are surrounded my myth and marketing FUD.
That’s why we took all the mystery & magic out of them by creating targeted attack vector blocking for known vulns. It is very simple, no magic, and like most simple solutions — it works well.
@ Arian:
Your conclusions contradict all the real world measurements I can demonstrate. The SQL Injection bot is a perfect example of where blackbox scanning + WAF integration solves for businesses issues cheaply.
I would never trust measurements solely from a vendor. If MITRE comes out and says that WhiteHatSec is the gold standard of modern metrics then hell will probably also freeze over that same day.
A perfect example? How? I just showed how it was a horrible example!
Out of the 640-some websites I measured during these attacks, only ONE input vulnerable to SQL Injection (out of tens of thousands attacked by this worm) was missed using a blackbox scanner. The Sentinel platform to be specific.
You can’t even start understand the problem if you think that it only has to do with you and your customers. 640 websites? Please. Yes, out of tens of thousands, that’s exactly the point.
The missed input was a “Blind SQL Injection” as many of these are. During the period of this worm we implemented a number of new syntax arguments that now successfully find the one missed issue
You can’t find the issue with a scanning tool or even a manual pen-test. The issue is lack of parameterized queries.
We also can create blocking rules for all of the issues we discover.
Seriously, Arian, please tell me that you don’t believe that! How could a WAF possible stop a business logic attack? How could a WAF stop second-order attacks? I think WAF’s don’t even stop the most common attacks.
A WAF prevents what the WAF vendors (and WhiteHatSec) want you to believe that they will prevent. Firewall vendors don’t go out and research how their firewall doesn’t stop certain kinds of packets under certain kinds of scenarios. They don’t explain the client-side attack vector (and the extremely high likelihood of it) to their potential customers.
WAF’s can block `known’ vulnerabilities, but in the case of “what a web application security scanner finds” — the results are not then “known vulnerabilities”, but instead should be thought of as “known software weaknesses”. Once a software weakness is exposed, this means that there are still many vulnerabilities yet to be uncovered.
Look, if web application security scanners were like regular black-box functional quality tools — they would see around 30 percent of the total bugs. Instead, they see 7, 8, maybe 10 percent (we must still have a lot of work ahead of us!). The only way quality testers were able to push their black-box tests above the 30 percent mark was to utilize pairwise testing as well as techniques that can manage keeping track of state-transitions. State-transition technology is not available in commercial web application security scanners, neither is pairwise testing. I’ve heard an argument that it’s impossible to even properly utilize pairwise testing for security property testing purposes. Even with these additional testing techniques, black-box functional testing with automated quality tools only rises to 70, maybe 80 percent coverage at maximum.
These claims on Jeremiah Grossman’s blog about 1 percent false positives in Sentinel are also wishful thinking. We all know that the false positive rate is highly dependent on the application in question. Let’s see how Sentinel fairs if I setup some apache configs (in an already huge and highly complex application) to respond over HTTP in varied ways (such as 200 OK to everything). There’s no rule of thumb for false positives and you can’t take in averages. Everything has to be customized.
In fact, this brings up a great point. WAF and WASS technology requires a seriously large investment in customization by an expert. We don’t have enough of these experts to go around. So do we start with WAF/WASS training, or do we instead just work with developers and train them in the first place — since they are the ones empowered enough to do something about a root-cause fix.
There are plenty of websites that use WAF that have been hit with SQLi or XSS. Some of these websites employ a large department of WAF and WASS experts. The adversary who found the SQLi or XSS typically is not an expert — often they are kids with 1 or 2 years programming experience, and with no other security experience — nobody is paying them — they are unskilled, unknowledgeable about web applications, etc. It is easy to find SQLi and XSS if you understand the code. It is easy to fix SQLi and XSS if you understand the code.
It’s difficult to use a WASS to find weaknesses or properly attack an application unless you customize it and you’re an expert. It’s even more difficult to deploy a WAF in prevention-mode, in order to prevent known vulnerabilities — let alone the root-cause issues which are systemic to a certain problem in the code. Why would we waste time on this approach? To keep WhiteHatSec and WAF vendors in business? Any other reason?
I could go back and get you the ratio of missed issues to inputs if you like. It’s probably 1 per hundreds of thousands of inputs; 1 per millions if I include known non-vulnerable code (to the bot)
Did you check for second-order injections? I guess not since Sentinel doesn’t even support them.
Even at 1 per 640 sites….that’s pretty good. That’s 639 winners in this lottery.
Wow, aspiring customers win a $110k WAF price-tag (plus $12K/year for support) combined with a $25K/year price-tag for a SaaS WASS solution! But, wait, they still are weak against the OWASP Top 10?! And what’s more is that adversaries are already preparing themselves for their next avenues of attack. Guess what? It’s still at the web application layer!
If someone else can actually demonstrate, and not pontificate, a better working solution…I’d sure love to see it demonstrated.
You are really opposed to the idea of fixing the code, aren’t you? It cuts into the big numbers on the income sheet and drops holes and bombs all over that business plan? Only if it wasn’t so difficult to get bought-out by a WAF vendor (since you couldn’t sell to an HP or IBM) — why they have to be having these kinds of issues at the same time??
I was anti-WAF too at one point, because they are surrounded my myth and marketing FUD.
I am still anti-WAF. Do you really think that’s going to change anytime soon? What is the problem you are trying to solve by posting this comment on our TS/SCI Security blog?
I enjoy the discussion; I love the quick wits — but do you really think you’re going to be able to convince me that concentrating all effect into fixing code is the wrong thing to do?
Hear me out for a second. Do you really think that they are ZERO quick wins that can be accomplished by involving the development groups in any given organization?
What about with specific regard to regression testing? I know that WhiteHatSec has customers that have been in regression testing for over 2 years. If you look at the WhiteHatSec numbers for Time To Fix, do you really think this applies to anyone EXCEPT WhiteHatSec customers?
To me, these numbers show what a horrible job that WhiteHatSec is doing. Maybe the problem is you?
I can think of dozens of ways to improve regression testing with regards to any sized development department — that would instantly result in a better understanding of reoccurring bugs. Within a few days, organizations get to see instant results if Continuous Integration (or some form of it) can gain ground and mindshare. With test-first development, bugs start to magically disappear, while at the same time — time in regression goes from months to hours.
Instead of forcing two things that go bad together into an abomination/monster — WAF and WASS, I’m trying to come up with ways of making other wins work well together, such as CI (continuous integration) and TDD (test-driven development).
That’s why we took all the mystery & magic out of them by creating targeted attack vector blocking for known vulns. It is very simple, no magic, and like most simple solutions — it works well.
Let me see how it works under the hood. It sounds simple on paper, but anyone who has worked with a WASS or WAF knows that it’s not. Even if WhiteHatSec was the best WASS solution (which it’s not), and even if WHS did hire the best talent then there would still be problems.
Arian, you are the best talent out there with regards to web application security. Jeremiah, Anurag, and at least 4 other people that I’ve spoken with at your company are also of extremely high caliber. But that’s 7 people, only one of which is really anything outstanding. 7 people and 640 websites is bad math.
The way that you focus your expertise and skills is just misguided. You should be working with developers, doing training, or getting the right ideas out there into the community by giving back.
If you’re not a part of the solution, you’re part of the problem.
DRE, I am not planning a war with you, clearly your attitude towards WAFs is representing your faith, and I never try to change someone’s faith… (I have found out it is almost impossible…).
I think it is important for your readers to also look at the Web security problem and solution from a pragmatic approach.
There isn’t a single solution for web application security, in real life noting can be 100% secure. Not even if you yourself will code all the web applications in the world, simply because you are a human, and humans make mistakes.
The real question is if WAF can significantly contribute to website security, the example that you used, the massive SQL injection attack is a perfect one, this attack was easily stopped by using a WAF, a WAF can protect you from a simple SQL injection attack or a blind SQL injection attack, it doesn’t really care.
Your assumption that the only solution to web application security is code fixes doesn’t stand in real life, you are missing the business problem aspect of this approach, you are also missing the challenges that enterprises have with this approach. Yes, if you have a blog and small web application that you maintain you could probably fix what you know about in the code, however for an enterprise, real life poses a few challenges.
1. The security officer is responsible for website security, however, he really doesn’t have any tools to enforce security, yes, he is going to the dev team and teaches them about security, yes, he is scanning the website and finds vulnerabilities, but how is making sure that the vulnerabilities are fixed? he doesn’t manage the dev team, the dev team will not get fired if the website was hacked, right? dev will always look at vulnerabilities like “bugs” and who doesn’t have bugs? on the other hand, who will get fired if the website was hacked? Who can’t sleep at night knowing that he has open 10 SQL injections vulnerabilities across his website that he needs to secure? WAF provide a way to raise dramatically the overall security of a website and provide the security officer a way to enforce it and sleep well during the night…
2. Do you know how many enterprises are not developing their web applications? (clue - every enterprise is using at least one web application that isn’t home grown… well, in real life, much more than one..) how will they secure them? many times, if they contact the vendor who coded them (hopefully the vendor is still in business) with a vulnerability report they discovered they get an answer, “yeah, we will fix that in the next release”, “oh”, “BTW, the next release is within 8 months”
3. What about the web server itself? should they also fix the code of the web server?
4. Patches, yes, patches are very important and everybody should deploy them, however, it takes time until vendor supplies patches to known vulnerabilities and it also takes time until an organization can schedule a maintenance window to apply these patches (oh and of course, test them before that…)
5. Logging and forensics, the logging which a web server is doing is very limited, what about logging of headers, POST data? you realize the benefit of such logging.
So thank you for raising the awareness for web application security issues, in the real life of enterprises, WAFs are becoming a major building block, like firewalls 15 years ago. Don’t get me wrong, I am not against fixing code, I just wanted to make sure your readers understand the challenges of enterprises and why they choose WAFs . By the way, I have to admit here that the company I work for, F5, is selling a WAF.
@ Ido Breger:
the example that you used, the massive SQL injection attack is a perfect one, this attack was easily stopped by using a WAF, a WAF can protect you from a simple SQL injection attack or a blind SQL injection attack, it doesn’t really care
Incorrect — the SQL injection attack (many different types of tools and threats were used) infected hundreds of thousands of web applications, which in turn infected thousands of Windows machines, turning them into a botnet. Although I expect that at least one or two F5 customers might have been protected against the second or third wave of these attacks, if they had a web application security expert configure their WAF properly.
However, I think those organizations would have been better saved by coding correctly in the first place, or even fixing the code when it popped up as an issue. They may have spent less time and resources — and they wouldn’t have to spend $110K for a WAF pair to do so.
however for an enterprise, real life poses a few challenges
Can you explain what those challenges are? I see WAF installation (configuration, verification, troubleshooting), operations, maintenance, etc as much more challenging than fixing the code.
WAF provide a way to raise dramatically the overall security of a website and provide the security officer a way to enforce it […]
What about the risks to the web application that the WAF does not protect against? What about the vulnerabilities that it cannot enforce? Millions of web applications are vulnerable to XSS, including every single one that sits behind a blocking WAF. Many XSS vectors go past the WAF detection modes as well, removing the need for a security officer to be woken up during the night. I don’t see this as beneficial.
Do you know how many enterprises are not developing their web applications?
I think that I have a pretty good idea having worked with many enterprises for a very long period of time, specifically regarding web applications and their vulnerabilities. When companies acquire applications (or software such as third-party libraries, components, frameworks, et al), they have to consider the risks. Managing those risks can be done in numerous ways.
how will they secure them?
“Test and verify before you buy / acquire”. For example by using third-party full-knowledge vulnerability assessment services, audits, evaluations, rating systems, assurance levels, et al.
many times, if they contact the vendor who coded them (hopefully the vendor is still in business) with a vulnerability report they discovered they get an answer, “yeah, we will fix that in the next release”, “oh”, “BTW, the next release is within 8 months”
It doesn’t sound like you’ve done this before. Have you worked as a vulnerability assessor? As a risk assessor? Certainly there are problems such as you mention (people do say those things), but there are ways to overcome them.
What about the web server itself? should they also fix the code of the web server?
I’m not talking about web servers. I’m talking about web applications. There is a very big distinction between the two. If you work for a WAF vendor, I suggest that you read up on the differences. You seem to mix the terminology as if they are one in the same.
Patches, yes, patches are very important and everybody should deploy them, however, it takes time until vendor supplies patches to known vulnerabilities and it also takes time until an organization can schedule a maintenance window to apply these patches (oh and of course, test them before that…)
Patches are what you install on the Operating Systems for the desktops and laptops that access web applications over a network. Patches prevent “client-side exploits”, and they often do not need to be installed right away if there are other controls in place to support a proper vulnerability management program. However, this again — has nothing to do with web applications. You seem to bounce around a lot, off-topic from the original conversation. I do understand why client-side exploits are relevant to this conversation about the SQL injection attacks (the botnet malware), but I fail to see how this has anything to do with WAF’s.
Logging and forensics, the logging which a web server is doing is very limited
Ok, so this is where tcpdump or socket or system-call tracing/logging comes in useful (or maybe a different web server). I’ve been using these types of tools for almost 15 years. Why do I need a WAF to do this?
in the real life of enterprises, WAFs are becoming a major building block, like firewalls 15 years ago
If this is true, I find it sad. Enterprises already have enough problems with respect to security. Adding a WAF into the operational picture must be a nightmare and a curse for these organizations. I don’t really relate the firewall and WAF technologies. First of all, firewalls are not as important as they once were because attacks come at the client-side through crimeware or other malware threats. Secondly, firewalls actively blocked network traffic. If a WAF blocked traffic, then I wouldn’t be able to go to the websites I like to go to. What do WAF’s block besides legitimate traffic and WAF vendor test cases? Do they block actual attacks? What kind of attacks do they fail to block? My guess is that WAF’s fail to block many, many, many kinds of web application vulnerabilities. We have a blog post about this called, Web application firewalls: a change of heart that you should definitely check out. It talks about a variety of even stupid check list stuff that WAF’s don’t block.
Hey DRE,
Looking at your posts all over the internet,I have a feeling that you belong to the “Anti WAF” religion, as such, I respect your religion. I will not try to convert you, I don’t believe in converting someone’s religion. The “anti WAF” religion is really a harmless small religion which helps to raise the awareness of web application security, it also actually drives pragmatic, down to earth security officers in enterprises to deploy a WAF solution which helps them solve a business problem. I will say it again, I am not against fixing code, its just that in reality, in real life, in large organizations, it has limitations which are preventing it from solving the problem.
This will be my last comment here, Your readers I believe, got the message.
Thank you again for raising the awareness around the subject that is so close to our heart - this is really important and you are doing a good job!
Cheers,
Ido
@ Ido Breger:
I would be talking about how great WAF’s were if they worked as advertised.
@DRE. Ah, hadn’t been back in a while. So you asked why did I post? Only to see if you were open minded. And you’re not.
It’s easy to deconstruct a solution, syllogism, etc., and you clearly delight in that. That doesn’t change the fact that I’m one of the only people on the planet with real, hard numbers and measurements. Sure, those measurements are from “whitehat customers” but it’s the best current sampling on the planet. (debateable, of course)
Anyway, I’m working hard to give people useful tools and solutions to solve their daily problems. I think you are completely out of touch with business realities. I’ve heard your kind of rants before from people who lack ecommerce business backgrounds.
So this will be the last comment I see myself posting on your blog. We mostly agree on what the problem is, however, at this point I am going to agree that we disagree, fundamentally, about how to approach the problem. I don’t see that changing unless you give me something pragmatic I can measure.
You of all people should praise the fact that WhiteHat gives people real, hard numbers they can use to learn, attack, criticize us, whatever. No one else is taking that step.
I’ve written courseware and I’ve trained developers and I’ve implemented SDLC programs and all that bullshit for more years that most of the pundits talking about this crap today. Touting that as the end game is a bunch of academic nonsense. You could of course prove me wrong with some relevant, modern measurements. I want both knowledge-based & skills-based measurements, and qualitative and quantitative analysis of defect reduction and quality improvements.
And then that turns into its own circular game. Go read The Mythical Man-month, or some of Spolky’s essays on quality improvements. meh.