Nah. Developers don’t need to know anything to make this sort of thing happen. All they need to know is that they wrote a bug and it’s considered an error. And that they should be writing unit tests that assert for that particular bug’s behavior. Once you teach them these things — the rest will be self-taught or easily preached.

In other words, I wouldn’t go slow. I wouldn’t wait to have Ounce or Fortify SCA as part of the system integration nightly builds. I wouldn’t wait to mark at least the top twenty or so vulnerabilities that apply to any one project as errors.

I also wouldn’t wait to get Fagan inspection working throughout the development culture of any organization. Then I’d combine Klocwork K7 along with MITRE CAPEC and CWE know-how for any given moderator of any given project.

WebScarab? I never even mentioned that. Giving WebScarab to a developer or quality engineer is a complete waste of time and money. Even WebGoat is a waste, in my personal opinion.

Developers need to learn how they already learn. You can’t throw any of this stuff at them. You integrate it silently; and when they start complaining — you throw in the idea of a reward system for programmers who can write good, reusable unit tests to find software weaknesses. For the programmers that continually make the same security mistakes over and over — it’s best to penalize/punish them until they can get up to par with the rest of the team.

If putting Armorize, Fortify SCA, or Ounce into the build server or integration process is overwhelming developers, then turn down the sensitivity. While I agree that requirements and UML aren’t required/necessary for every project — they should certainly take up 10% of the time and budget of any project — and security should be a part of that process, especially looking at criteria such as MITRE CAPEC, attack-modeling, and Application Architectural Risk Analysis.

The most difficult part of my CPSL is the verification of fuzzing/fault-injection tools that are run during integration, but I provide ways of making this easier. Some of these might run very clean assuming how these compare to the continuous-prevention unit tests that the developers have used throughout the process. It’s up to everyone to make improvements; not just the CSO and his/her team.

Get and install WebScarab, then after a while get something plugged into ypur IDE, then really by that time they will start to realize what they need from this list of resources. I’d be careful to not overwhelm them.

I’m glad you are looking at ESAPI, because that is a CSO’s dream and there aren’t enough people using it.

You certainly suggest a lot of tools, and in a dream world we’d be able to utilize all of them. Unfortunately, assessment times are usually scoped fairly tight, internally or externally.

I don’t think people should be using all these tools. I’m recommending some based on an order of preference. This ordered list is mostly for learning what the tools are capable of for assessment work.

In the case of doing a vulnerability assessment on a large web application — I could point you towards a world full of resources. It’s not just tools, but I happen to be talking about those here.

As I said in the first post of this thread, “My feeling is that vulnerability assessments are typically done less strategically/operationally in IT environments (relying too much on tools and point-and-click scanners), while not hands-on enough for IT dev shops (or unknown where to start)”.

Short answer: personally, if speed is important; I would use JSView and FireBug for DOM-based XSS findings if source code wasn’t available. If it was, I’d use Fortify SCA and a combination of open-source and commercial tools that would be largely language dependent (Milk, LAPSE, Orizon, PMD, FindBugs, jCUTE for Java; PSA3, PhpSecAudit, Inspekt, Pixy, PHP-SAT, PFF for PHP; ASP-Auditor, XSS-Detect, PREsharp, FxCop, Compuware SecurityChecker for .NET; DN_BOFinder, PREfast, PREfix, AppVerif, CUTE, Compuware SecurityChecker for Classic ASP; plus RATS, SWAAT, AppPrint, AppCodeScan, AppScan DE, and DevInspect for most languages).

I know that’s not a short answer, but wait until you see the long answer, below.

I don’t think many people would be able to use a lot of these tools and still get maximum coverage in other areas like authentication, access control, etc. What can you say about prioritizing toolsets in a real world assessment?

When testing a large application, I would use my CPSL methodology, if at all possible.

The recommended toolsets have an equal priority (unless they are CWE-Effective or CWE-Compatible). AFAIK, I recommend 5 kinds of tools
1) A requirements analysis tool such as IBM Rational Requisite Pro or the free tool from NASA SATC’s ARM
2) Secure design inspection tools that work with UML 2.0, such as IBM Rational Rose or better Klocwork K7. There is also an open-source tool called Fujaba that can be used to move between UML and Java source code
3) A CWE-Compatible static analysis tool. Right now, only Veracode would fit into this category (so, clearly, they would be prioritized first), but aspiring CWE-Compatible tools such as Armorize, Klocwork K7, GrammaTech CodeSonar, Ounce Labs, and Fortify SCA would fit into this category. Milk/Orizon/LAPSE/PMD/FindBugs or ASP-Auditor/XSS-Detect/SWAAT/PREsharp/FxCop could be FOSS replacements for the above as long as CWE is customized into the tools
4) Manual secure code review with a workflow tool such as Atlassian Crucible, although Trac could be a FOSS alternative
5) Final verification with another CWE-Compatible tool that is capable of fuzzing/fault-injection for integration testing. No tools are currently CWE-Compatible and allow for integration testing (however I would like to see this change). The best tools are FOSS tools such as CUTE, Canoo WebTest, or Windmill that would be modified to support CWE and CAPEC testing. Alternatives in the commercial world include AppScan DE or DevInspect — although these are difficult to automate for continuous integration when they are built into the IDE — and they also integrate less well with unit testing. One of the more interesting tools that I’ve come across lately for this purpose is UTScapy (although built more for protocol dissection and packet mangling than for web application testing, although still valid for modern application testing)

My views on vulnerability assessments are quite different, as you can see. The rest of the methodology outside of the tools is also important. There are a few enhancements that I’m working on to the CPSL process, such as integrating ideas from OWASP ESAPI as well as work I’ve done on the OWASP Web Security Certification Framework.

The most important section of the CPSL methodology is the continuous-prevention part, where reusable unit tests that assert the behavior of known security-related bugs are used at build-time. It’s like regression testing, but better.

Not to mention, some tools have overlapping areas of functionality.

> Any claims that an automated scanning tool can find all XSS is
> dead wrong when it comes to DOM-based XSS (or Flash based
> XSS, et al).

Not just DOM-based XSS, even reflected and stored, especially in Web 2.0 applications. The scanners just don’t understand the technology well enough. And this message has to be evangelized loudly because right now the loudest people in the room are the scanner marketing companies.

Just yesterday we ran one of the big named automated scanners against a client’s Backbase application with at least 30% of the input fields being vulnerable to XSS and we got kaput from the scanner due to the Ajaxy nature of the app.

> Best DOM-based XSS attack tools
> Web Developer’s View Javascript, FireBug, w3af, Burp Intruder,
> scanajax, ProxMon, OWASP Pantera, cruiser, CSpider.js

You certainly suggest a lot of tools, and in a dream world we’d be able to utilize all of them. Unfortunately, assessment times are usually scoped fairly tight, internally or externally. I don’t think many people would be able to use a lot of these tools and still get maximum coverage in other areas like authentication, access control, etc. What can you say about prioritizing toolsets in a real world assessment?