2007 Security Testing tools in review
In my last post, I explored some ways of using formal method tools to perform security testing in the most advanced scenarios. It may have been over the heads of many people, so I wanted to offset that by talking to some basic tools which I think anyone can utilize effectively assuming they bring the most important tool when security testing: their brain.
Of course, you need to think like an attacker, and saying that, I can't recommend a better source than CAPEC and their outreach and enhancement documents. Books that come highly recommended (read at least one of these) include:
- `The Web Application Hacker's Handbook <http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470170778.html>`_
- `The Art of Software Security Assessment <http://www.aw-bc.com/catalog/academic/product/0,1144,0321444426,00.html>`_
- `Hunting Security Bugs <http://www.microsoft.com/MSPress/books/8485.aspx>`_
- `Security Power Tools <http://www.oreilly.com/catalog/9780596009632/>`_
I really think that having some background in development practices is also necessary, so I recommend other books such as:
- `Continuous Integration: Improving Software Quality and Reducing Risk <http://www.aw-bc.com/catalog/academic/product/0,1144,0321336380,00.html>`_
- *Secure Programming with Static Analysis*
- `Software Security: Building Security In <http://www.aw-bc.com/catalog/academic/product/0,1144,0321356705,00.html>`_
- `The Security Development Lifecycle <http://www.microsoft.com/mspress/books/8753.aspx>`_
If you want to write or develop exploits in addition to finding security bugs there are a few must have books such as:
- `Fuzzing: Brute Force Vulnerability Discovery <http://www.aw-bc.com/catalog/academic/product/0,1144,0321446119,00.html>`_ ` <http://www.wiley.com/WileyCDA/WileyTitle/productCd-047008023X.html>`_
- `The Shellcoder's Handbook <http://www.wiley.com/WileyCDA/WileyTitle/productCd-047008023X.html>`_ (2nd Edition)
- *Real World Fuzzing*, Charlie Miller, Toorcon 9 (to be included in a future book)
- `Writing Security Tools and Exploits <http://www.syngress.com/catalog/?pid=3620>`_
If you have little operations background and want to learn how modern-day systems and networks are protected, I highly recommend:
- `Network Security Hacks <http://www.oreilly.com/catalog/netsechacks/>`_ (2nd Edition)
- `Virtual Honeypots: From Botnet Tracking to Intrusion Detection <http://www.aw-bc.com/catalog/academic/product/0,1144,0321336321,00.html>`_
- `Preventing Web Attacks with Apache <http://www.aw-bc.com/catalog/academic/product/0,1144,0321321286,00.html>`_
These books will introduce you to hundreds of tools - many of which are useless, difficult to configure, difficult to understand, and/or about as robust as the software you're trying to break. While learning some background and technique is useful, by and large I think you'll get the hang of a few tools that do one thing well and stick with them.
The most important and widely used security testing tool starts with program understanding / comprehension. If you have source code for the application, SciTE is an excellent source code editor that can be scripted using a language that will appear throughout this blog entry: Python. Probably the best way to learn Python is to code Python to help you code Python using SciTE. This will also introduce you to recursion and meta-level concepts. However, you don't need to become an expert in every programming language to use SciTE for secure code review -- all you need to do is learn how to perform a manual DFA (or CFA) for security weaknesses on each language one-time, and then automate that DFA/CFA through Python scripts in SciTE.
Before manual code review, developers may have created program specifications. In the last post I mentioned formal top-level specifications, which is rare. More common may be UML models or DSD's. Using open-source tools such as Fujaba, which can move between UML models and Java code in Eclipse, may prove useful. Ideally, this would be done using a formal UML tool such as the commercial Rational Rose, or better -- one specifically for security design inspection, such as Klocwork K7. There aren't any free or open-source tools that are specifically geared towards secure design inspection, but expect this to change in the near future. The best work I've seen lately has been done by the Octotrike team, also recently spoke to at Toorcon 9 as Privilege-Centric Security Analysis.
As most security testers will come to understand, not all program source is easily available. For .NET assemblies (CLR), try using the .NET Reflector; Java can use Jode; and Flash can use Flasm or Flare. There are some C and other compiled-language decompilers, but consider using a debugger for your language if it becomes too complex to easily decompile (or too inaccurate). In other words, use a debugger such as the basicDebugView under Windows (for basic stuff), JSwat for Java on any platform, ImmDbg for advanced Windows debugging, and IDA Pro (for as many platforms as you own and have access to) if you really want to rock.
IDA Pro, while being both an excellent debugger and disassembler, is a cut above the rest, although I could name plenty of other tools in its class. IDAPython is the language of choice for such a tool, bridging over to Immunity Debugger and several other tools. There is a free version of IDA Pro, but any serious person will shell out the money for the non-free version if they value the primary tool in their toolchain.
Viewing code and debugging code will help with program flow and understanding. Some Python tools such as HeapDraw and Python Call Graph make great visualization tools to aid in this type of work. You'll want to learn how to implement automated DFA (data flow analysis), view control flow graphs, basic block graphs, and how to view these under a scenario called hit-tracing. Hit-tracing will allow you to watch the flow of the application only under certain conditions that you want to see, while filtering out everything else. The PaiMei tool will provide this, and by doing so - you're really discovering an important aspect of code known as coverage; how much the testing affects the code it runs through.
Normally developers utilize code coverage during a recent invention of quality testing made specifically for developers by developers called "unit testing". Unit tests are quick 1-3 second checks in their IDE that they can use to assert behavior and provide an instant fix. More advanced unit tests called "component tests" can be done during a build, along with mocks (fake database or other system necessary to assert behavior of a fully implemented application). Typically these are all for functional tests, and possibly also for performance. Rarely are applications tested for security in this manner, and unit testing was really built for functional tests (although they theoretically could be adapted for security testing).
It's sometimes ok to pretend you're a developer just to see what idiotic things they skip over when they write and build their code. In this case, you will want to load the code into its native IDE (say, Eclipse for Java). Then, promote all the warnings you can to errors, build, and see what the environment spits out at you. Load automated static bytecode analyzers, which often have a large amount of security checks (e.g. FindBugs, FxCop, CAT.NET, and AspectCheck), as well as source code checkers -- and use this time to write some of your own basic checks (especially to find low-hanging fruit). For Java, PMD is extensible through XPath, for an example see the PMD SQLi Rules. Also check what demos are available in terms of CWE-Compatible tools - I've used Fortify SCA to great effect here. There are also plenty of open-source static analysis tools, but many aren't as complete as their CWE-Compatible commercial cousins. If you can find some basic unit tests to run (e.g. Crap4J), these may also provide better program understanding, especially when combined with inspection tools such as Armorize CodeSecure, Fortify SCA, OunceLabs, Klocwork K7, GrammaTech CodeSonar, and the bytecode checkers (i.e. FindBugs, FxCop, CAT.NET, and AspectCheck).
Security testers have created their own sort of unit tests which involve injecting faults into an application, or even better -- sending random or specially-crafted data to the application inputs -- often called fuzzing or `fuzz testing'. Fuzz testing tools are numerous, although one of the best tools is EFS, which stands for Evolutionary Fuzzing System. EFS provides random data to an application along with code coverage reports from PaiMei. It enhances the tests using this code coverage data by implementing genetic algorithms. Some of these tests cannot be solved, but a satisfiability solver can provide extensive coverage beyond GA's in this situation. STP, zChaff, and Saturn are examples of satisfiability solvers, while catchconv integrates these concepts as a variant to Valgrind (a popular dynamic analysis tool, similar to IDA Pro, but focused mostly on memory debugging). Valgrind's Memcheck, DynamoRIO, Purify (commercial), DynInst, and Pin are actually dynamic binary instrumention (DBI) tools which would work well with PaiMei's crash binning routines when attempting to find off-by-one's and incrementing counter crashes (and other issues that may be better found through formal method security testing).
Random fuzzing along with code coverage is probably one of the most advanced tools in a security tester's toolbox. However, there are some cases where binary protocols hit protocol parsers on both sides of a connection. Tools such as ProxyFuzz and Universal Hooker can make it easy to determine whether one side or both sides of a connection require more than just random fuzz testing. If this is the case, a lot of manual work may be required to determine which parts of the data are encrypted (if any), compressed (if at all), or separated by TLV's (type, length, or values). Using a specification may be the easiest way to implement a specially-crafted fuzz testing tool for the "application under test" (AUT). Using a fuzzing framework such as Sulley or Peach can make building this tool much easier than doing so by hand. In some cases, files (not protocols) are loaded or unloaded by applications. Certain fuzz testing tools work specifically well for writing random or specially-crafted files, and others are provided by frameworks. Depending on the platform or type of application you're targeting, you'll want to look at a few different file fuzzing tools/techniques, but FileFuzz (by Mike Sutton) seems to stand out as a good starting point (similar to ProxyFuzz for protocols). For other file fuzzing tools, see FileP (my favorite, written by Tim Newsham), SPIKEfile and notSPIKEfile, Ufuz3, FuzzyFiles, and Mangle. I'll leave targeting browsers, web servers, and web applications with fuzz testing techniques for later, but these should give you a general idea of what is available. There's also fuzzing environmental variables, in-memory, etc. Note that Sulley, Peach, and FileP are all Python frameworks for coding specially-crafted fuzz testing tools.
Often there are situations where rolling your own fuzz testing framework or dissecting a protocol or file format may seem like the last thing you want to do. Prototyping this work should be a first step to determine if the effort is worthwhile. I gave examples of using both ProxyFuzz and uhooker as starting points. Combine this with code coverage, and check to see if the results show how well you've tested. Before starting a fuzz testing project using a framework or writing your own tool (or a full-blown client), you may want to do a few more checks. This is where knowledge of bash and C can come in as extremely helpful. Small C programs are fast, easy to write if you're just trying to do one thing well (run a looped test), and can be easily piped on the Unix command line to "try different things out". The bash shell provides ease of use when writing loops or conditionals for a bunch of small programs piped together, to combine speed with agility and flexibility in prototyping. Using the Matasano Blackbag tools (or by writing your own tools similar in nature/effect), it becomes easier (and more fun) to reverse binary protocols/formats than by reading specs and building run-once Python scripts. Of course, finding the Matasano Blackbag tools may be difficult as they now recommend writing protocol dissectors using Ruby along with BitStruct. A commenter posts using Construct with Python, as well as the popular scapy Python library found quite often in the literature.
Almost a lost art, but related to file fuzzing and binary protocol dissection is binary analysis. Many continue to use IDA Pro to perform these sorts of bug-finding techniques, using IDC scripts such as the one by Halvar Flake, BugScam. Tools such as bugreport, BEAST and OBOE can also be used, but it may be better to re-write this functionality in IDAPython, possibly by using x86 emulators, or PEiD to remove potential code obfuscations. Halvar Flake's company, SABRE Security, also maintains the best tools for program analysis, debugging, and testing for security. The primary reference tool, BinNavi, works on embedded processors as well as most platforms you can think of. I've heard he can make it available to people who cannot afford it depending on your project scope. BinNavi/BinDiff have simply the best reputation in the business, and like PaiMei, require IDA Pro.
After a security-related bug is found (using automated inspection, automated fault-injection, automated random fuzzing, or by partially automating specially-crafted fuzz testing, binary analysis, or code review), some will choose to report it to vendors, some will write exploits, and some will do none of the above. If an exploit is intended to be written, you'll find lots of support for doing so using the Immunity Debugger and the Metasploit package. There are some automated exploit writers including the MSF-eXploit Builder, Byakugan/noxdbg, and Prototype-8. Python appears to be dominant for exploit writing were it not for the Metasploit project, which has moved to Ruby. If the focus of a security testing team is dominant towards writing exploits, then it is suggested that Ruby equivalents replace SciTE, ImmDbg, IDAPython, PaiMei, and the various fuzzing frameworks and tools already mentioned, or that security testers learn both Python and Ruby equally well.
In fact, finding web application vulnerabilities is often more than just program understanding along with standard fault-injection and fuzz testing. There are many intricacies to finding XSS and SQLi -- even other critical vulnerabilities such as HTTP Splitting/Smuggling can take many forms (see: Amit Klein on HRS). Fortunately, many of these are covered in The Web Application Hacker's Handbook, in fairly clever detail. The book even covers triple-encoded XSS attacks, second-order SQLi, OOB SQLi, and inference attacks. Almost all examples use the Burp Suite, as the primary author (Portswigger) also wrote that tool. The book makes special mention of some other point-tools, including the aforementioned inference attacks using Absinthe. SQLi can also benefit from non-standard methods such as using PQL.
Earlier in this entry, I mentioned some Firefox add-ons such as UrlParams. There exists a similar add-on that can be used as an HTTP editor (but only to modify requests, not responses): TamperData. While mentioned in Portswigger's book, he doesn't cover it as well as he should. TamperData can be opened in the sidebar and looks surprisingly similar to UrlParams -- this has great benefit, especially when testing for low-hanging fruit SQLi, XSS, and HRS vulnerabilities; as well as path traversal, logical flaws, URL redirection, and similar vulnerabilities which require direct tampering of the HTTP request headers. If you want to build advanced attacks, besides Hailstorm or Burp Scanner/Suite I can also recommend Wfuzz (written in Python) and CAL9000 (a browser driver, ideal for testing XSS in several different browsers using the same tool). Of course, there is the original HTTP/CGI attack tool, which has recently made a comeback as Nikto 2.
One tool is mentioned that I had not heard of before, and interestingly, the approaches given to attack authentication, session management, and access control are very different than the approaches that I propose (and the tools that I use). The CookieWatcher add-on is proposed as a way of looking for session ID's in applications under test. This is an excellent idea, and I've already integrated it into my testing strategies, as it is easy to change which session ID type you are looking for (it's the only tool option/preference, which has a drop-down menu that shows a list of cookies seen), delete, or copy (as well as view it in the status bar). While the lack of options is somewhat aggravating, the add-on works well along with CookieSwap (to have switchable cookie profiles), CookieSafe (to turn first and third party cookies off on a case-by-case basis), CookieCuller (to permanently protect cookies), and Add N Edit Cookies (to use the Cookie Editor to modify content, path, security, or expire information, as well as to add completely new cookies). WebScarab, Stompy, and the forthcoming Burp Sequencer are probably excellent tools for session ID analysis, and the book covers how to do this in detail, while pointing to the excellent NIST FIPS 140-2 standard, which has four levels of validation criteria. Interesting characteristics that may affect authentication or session ID randomization include IP address, BGP prefix, DNS forward or reverse entries, time, and HTTP Referer or User-Agent. Tools such as RefControl, User Agent Switcher, Tor, and pbounce should be used when testing for authentication (IP, DNS, and Referer authentication checks should also be configuration or source-code assisted if possible).
A lot of the functionality of the Burp Suite is utilized rather effectively in the book as well. However, the lack of inclusion of certain web application security aspects, such as Ajax security, automated DOM-based XSS scanning, and Web services security also happen to be problems with the Burp Suite itself. For Ajax crawling, I suggest checking out all the work done by Shreeraj Shah (especially Crawling Ajax-driven Web 2.0 Applications and Scanning Ajax for XSS entry points), including his new BlueInfy tools. For DOM-based XSS, the W3AF tool has a module to support this that is equivalent to Shreeraj's work. Web services security is also best covered by Shreeraj Shah, although there will be a few books coming out in the next month that should re-address the issues of Ajax and Web services rather well. From my experience, the OWASP Interceptor tool, as well as the commercial (free limited version) SOAPSonar tool are good starting points for Web services security testing outside of Shreeraj's tools already mentioned. There are fuzzers, including the OWASP WSFuzzer, SIFT, and iSecPartner's WSBang, as well as generic XML fuzzers such as untidy and Schemer.
It is strange that without mentioning much about Ajax, XML, or Web services that the authors included a very detailed section on JSON Hijacking and CSRF. While I was familiar with both CSRF dorks and CSRF Redirectors, the book contains excellent material on how to test (although does not provide a tool reference). Most recently, OWASP released a new project called CSRFTester, which looks promising. OWASP has a lot of great projects, but their integration (I agree with Ory Segal on this) could be vastly improved. There are very unique tools such as Pantera (Python extensible) and ProxMon (also Python extensible) that do passive analysis for web application security testing, which can save countless hours of manual penetration-testing using TamperData or Burp Suite. I often wish that other tools such as the THC releases and Foundstone free tools could also be combined, as I often reference these tools for the SSL checking support as well as a variety of other reasons.
For the year 2008, I'd like to see replacement of standbys like RSnake's XSS Cheat Sheet. While incredibly useful to me in 2006, it no longer holds its weight compared to the WASC Script Mapping project and tools such as .mario's PHP Charset Encoder, or Gareth Heyes' Hackvertor. One of the biggest lessons I learned about software weaknesses (SQLi and XSS in particular) is that there is a concept of low-hanging fruit (LHF), but at some point after the root-cause is found, more complex attacks often work. However, this is largely true only when a security tester has full source-code and framework knowledge. In this instance, a spreadsheet such as the one found in the Microsoft Press' Hunting Security Bugs companion content, "Reference -- ASP.NET Control Encoding", can be used to figure out which classes encode on a per HTML, Script, or URI basis. What I've been describing for awhile now is what will be known as hybrid analysis, with hybrid (static + dynamic) tools becoming more rapidly available to security testers.
Speaking to browsers, I promised some information on fuzzing browsers (to include ActiveX as well). First of all, we must mention Michal Zalewski, who not only recently brought us Bunny The Fuzzer, but also started a modern version of the Crusades to create a battle between browsers and web applications when he wrote MangleMe. hdm followed to create a bunch of tools: Hamachi, CSSDIE, DOM-Hanoi, and AxMan (for ActiveX similar to iDefense's COMRaider). Before these guys, PROTOS had c05-http-reply, and eEye was still thinking about TagBruteForcer. Even the Mozilla team released a tool, jsfuzzfun this year, including collaboratively with Opera. The latest comes from GNUCITIZEN as WEB CLIENT FUZZER.PY and Ronald as BrowserFry.
Robert Hansen (RSnake) recently spoke at OWASP AppSec 2007 about Web Browser (In)-Security (slides not available yet, so I don't know what he covered). It's true that 89% of security vulnerabilities in browser plug-ins from this year were in ActiveX applications. However, that doesn't mean that you shouldn't keep your browser, Adobe Flash Player, Java installation, Adobe Reader, QuickTime, Windows Media Player, and Mozilla extensions up-to-date at all times. It also doesn't necessarily mean that IE is more insecure than Firefox, Opera, or Safari. They're all insecure and it's your job to find out where and why. RSnake did release some very cool code this year with the Master Recon Tool (aka Mr. T)` <http://browserfry.0x000000.com>`_. Expect to see more available at Jay Beale's ClientVA website (and read his presentation from Toorcon 9 on that same page).
Outside of regular security testing at home or in the lab, I'd like to address the two commercial Software-as-a-Service (SaaS) solutions available as outside security testing augmentation. WhiteHat Security makes a product called Sentinel, which embodies the WASC Threat Classification (i.e. a way of understanding attack-paths against web applications). WhiteHat has a mature understanding of web application vulnerabilities from an attacker perspective, which is ideal for people that are learning how to think like an attacker. Veracode also provides a service, SecurityReview, which also happens to be CWE-Compatible (and the only commercial solution that has to-date formally passed the criteria besides SofCheck, an Ada source code analyzer). CWE is invaluable information that can be given back to developers in order to fix security vulnerabilities and avoid software weaknesses in coding efforts.