Maybe I’m too romantic - I just don’t see someone who follows instructions well with an extremely detailed, multi-angle test plan doing as good a job as an experienced, organized hacker. Deviation, adaptation and creativity are what really allows attacks to happen. Too simple, maybe, but I think there’s truth in it

I think that experience is also best. It’s the brain behind the tools, not the tools that count most. However, there are good testing tools and bad testing tools. There are good test cases and bad test cases. There are good testing methodologies/processes and there are poor ones.

The CPSL guide mentioned specific tools, but it did not enumerate them in a list. I casually made references to them. You should also note that the CPSL is only informally documented on this blog and nowhere else in a very conversationalist style manner with excellent commentary such as Ariel’s and yours.

First of all, I want to say that “checklists save lives”. If it’s good enough to do the most difficult triage in an ER, and it’s good enough to keep people alive in IC units - then I’m sure there is just something to humans using good tools with good checklists.

MITRE CAPEC and CWE are our checklists. MITRE CWE Vulnerability Theory can turn a newbie into a professional with a little passion and some intense involvement in a weekend. Hand a Java programmer a copy of Secure Programming with Static Analysis, TAOSSA, and a bottle of tequila — then watch the vulns fly.

We’ve been doing non-standardized, ad-hoc testing for too long. What some people have created is a sort of movement towards a standard, informal method of security inspection/testing. We need some people using formal methods for security. We need lots of people performing secure SDLC work, preferably something closer to my CPSL than to the Micrsoft SDL.

Of course we’ll still need new vulnerability research done by new vulnerability researchers. It should be a little different than in the past, and it already basically is if you look around our industry.

This isn’t a new argument, there has been controversy over software testing for at least 20 years now. I think exploratory testing is best done with all the knowledge possible and the best tools.

What security professionals are missing is Vulnerability Theory, and they need to start with a Glossary of Vulnerability Testing Terminology.

They’re also missing out on some of the best tools. When Eugene Kapersky noticed that Peter Szor was using manual renaming of every variable in 1997, he introduced Peter to IDA Pro, which supported automated cross-references. Peter’s methodology was flawless, but he didn’t know or wasn’t using the latest in technology for structural analysis.

Right now all the best tools are language/bytecode-specific — best example is .NET Reflector and how it rocks all over IDA Pro. However, there are even more choice examples that would affect the CPSL. For example, testing tools are now being built into the frameworks such as WebTest in Grails. Defenses are built baked into frameworks such as HDIV. Certain techniques allow “after-the-fact” upgrades such as AOP weaves for code instrumentation and dependency injection for design (only supported well in Spring MVC today).

I think I’m arguing against, “A fool with a tool is still a fool”, but I’m not sure yet.

> Often, I see penetration-testing or security companies or
> penetration-testing security vendors doing what I consider to
> be slightly more useful than commercial AV or overly expensive
> firewall appliances.

I haven’t interacted with all of the companies but I know that Aspect gives its clients an unbelievable amount of useful information on how to break and how to fix without FUD and useless false positives.

> This is part of what I’m trying to say here. As for an “execution
> plan”… it appears that both Ariel and I have given some pretty
> heavy plans for all angles of software assurance and
> penetration-testing (to include cryptanalysis). Did you miss
> those, because I can point you towards them?

Maybe I’m too romantic - I just don’t see someone who follows instructions well with an extremely detailed, multi-angle test plan doing as good a job as an experienced, organized hacker. Deviation, adaptation and creativity are what really allows attacks to happen. Too simple, maybe, but I think there’s truth in it.

> and for those people to take a hint and get out of the software
> assurance / security industry before they don’t have jobs just
> like the AV people.

Whoa, reverse-giving-it-back-to-the-security-vendors FUD! I like it. =P

> I’m not exactly sure what you mean here. If I hear you
> correctly, you’re saying that the only way to be scientific is to
> help customers build more secure code.

What I was saying is that metrics, assurance, SDLC improvement, security processes - that is where the repeatable, demonstrable science really can be.

Also, email me, I’ve got news about browser group. Well, maybe not news. Just email me!

Often, I see penetration-testing or security companies or penetration-testing security vendors doing what I consider to be slightly more useful than commercial AV or overly expensive firewall appliances.

The presupposition of this thread is that pen-testing is dead / pen-testing doesn’t matter… whether software or network.

What I’m trying to say here is that a model like the CPSL can completely replace the need/desire to pen-test by security professionals — and for those people to take a hint and get out of the software assurance / security industry before they don’t have jobs just like the AV people.

If pen-testers still want to pretend that they are 17 yro haxors and “legally break-in” to Fortune 1000, then they can do it under the auspice of compliance.

From what I understand, Aspect Security does both PCI review as well as “real” security review. So you should be familiar with the difference here.

The recommendations, cleanup and the overall process of helping the client build more secure code are what should be demonstrably scientific. Fortunately, you can never lay out an “execution plan” for a great piece of art.

I’m not exactly sure what you mean here. If I hear you correctly, you’re saying that the only way to be scientific is to help customers build more secure code. This is part of what I’m trying to say here. As for an “execution plan”… it appears that both Ariel and I have given some pretty heavy plans for all angles of software assurance and penetration-testing (to include cryptanalysis). Did you miss those, because I can point you towards them?

I think most clients fundamentally don’t understand what hackers do and how they think.

I think clients need to have a security program that takes into account both application and software security. In my post on Building a security plan, I discuss possible frameworks for building a security program, as well as risk analysis for applications/software, vulnerability management for application security, vulnerability theory to understand how hackers think (which provides more details than most hackers know how they think themselves), and software assurance practices, as well as how they relate to the customer’s customer.

I think you have a very narrow view on my approaches, although not as quite as narrow as most of the rest of the industry — who usually outright disagrees with what I’m saying in this sort of anti-pen-test rant.

I always make sure coverage is good across all the security areas like authentication, access control, xss, csrf, etc., but complicated pen testing structures are only useful for those people don’t get the art of hacking. I’m glad there are other people who see the intersection.

The recommendations, cleanup and the overall process of helping the client build more secure code are what should be demonstrably scientific. Fortunately, you can never lay out an “execution plan” for a great piece of art.

This is a discussion that people should have more as I think most clients fundamentally don’t understand what hackers do and how they think.

I understand the difference and I agree on your take. There’s a lot of good research to work over. All the constructs you mention have really helped to secure our systems.

Notice, that there’s a better analysis in securing systems than in finding bugs or breaking them. It seems to me that one needs to be proficient in the latter to master the former.

I didn’t mean that the crypto models I mentioned could apply to a broader scope. In fact, I don’t think so. Also, they are aimed at proving security properties of systems, but not at finding problems in them.

Cheers,
Ariel

In information security, pen-testing refers to breaking into specific networks or computers (i.e. Application or Network Pen-Testing). In vulnerability research or software assurance, it refers to finding security-related bugs in software (i.e. Software Pen-Testing, Cryptanalysis).

However, we are faced with a problem that is very difficult to solve: designing a good security model

For systems, we have learned to mold concepts together. For example: Samhain, chkrootkit, rkhunter, the99lb, DieHard, PaX, chroot, improvements to GCC (userland) + St. Jude / St. Michael, DSI/DigSig, grsecurity (kernel) + protection from virtualization rootkits + detection of network covert channels + managed SIEM, etc. No formal methods here, but we could start by eliciting their designs, proving some of them, and model-checking their sources.

The same is not yet true for applications, which suffer from similar problems that systems have gone through (with very few sandboxes, trusted paths, logging, thresholding, monitoring, etc). Jeff Williams is working on a project called ESAPI which works to solve some of these issues for web applications. Modern applications of all types need higher functional protections and assurance levels. OWASP is also starting a Browser Security Project as well to solve the “other” side of this problem.

For defining access control matricies - RBAC is better than closed DAC is better than open DAC is better than programmatic controls. An even better method would be to use declarative controls, whereby application components have access to different databases with varying levels of database user privilege depending on application user group access rights.

I checked out some of the crypto threat models you mentioned, and while they appear interesting for solving crypto design and crypto protocol implementation security models, I’ll have to analyze them further to see if there is any value to other components in secure software design review.

Clearly you know exactly what I’m referring to with all of this since your presentation on ND2DB is exactly the type of research in software assurance that needs to happen.

Great. This work you mention can only benefit us (and many thanks for the pointers!).

It seems to me that webapp testing tools comprising fuzzers et al. are still young and have a lot ahead, static analysis tools are further in the way (they grow on several years of work) and these two are catching up with security requirements. As you say, we are being more competent at specifying these requirements and describing threats. It is only sensible that these two meet.

We are becoming better at assessing certain security properties. For these, tests are necessary. As you mention, we are combining these tests intelligently. Good!

However, we are faced with a problem that is very difficult to solve: designing a good security model. We can build tests for certain known attacks, but we aim to prove that our systems escape all attacks. Even if we got automated theorem proving right for a security model (which I don’t think it is feasible), we’d need to come up with the right models. For example, crypto security models (e.g., Canetti, Dolev-Dwork-Naor, …) do not involve side channels (e.g., a system can be called secure under a security model and still be vulnerable to a timing attack).

Nobody said it would be easy ;)

I want to see software assurance (SwA) tools take CAPEC/WASC and CWE/OWASP-T10 as input or send as output. Net/app pen-testing tools can take CVE as input or send as output. CVE is known vulnerabilities while CWE is known software weaknesses and CAPEC is known attack-paths.

Some pen-test scanning tools already combine these concepts (e.g. ones made by CORE, ImmunitySec, Rapid7, Foundstone), while others will be adding SwA support very soon (Qualys, W3AF, Tenable, nCircle). I’m not sure we’re going to see CWE-Compatible tools add support for net/app pen-testing. Similarly, I don’t think that CVE-based scanners are ever going to be CWE-Compatible or CWE-Effective.

Thanks for the discussion and pointers towards those documents. I agree that net/app pen-testing certainly have elements of science (also: I didn’t intentionally mean to mix the concepts). However, I was referring to SwA pen-testing (fuzzing or fault-injection), which are largely ad-hoc testing with little to no informal arguments or proofs. Just as you said, even crypto systems rarely undergo any formal, semi-formal, or informal method specification or testing.

As for the automated vs. manual testing: I agree. You’ll see in my CPSL above that Fagan inspection, manual code review, and manual verification of exploitables are required. What differentiates my CPSL from any other secure lifecycle is that the checks are turned into unit tests - which provides a process improvement sans type I/II errors.

I think to some degree - anything can be automated, even if you’re just using software to improve or speed-up workflow. This is why I listed some tools such as Klocwork K7 and Atlassian Crucible. Crucible doesn’t make it so that Fagan inspection removes the manual work, but rather makes the process more formal and improves the time to deliver results.

As for heuristics and coverage in fuzz or fault-injection testing tools, you are correct that not all measures are sound (e.g. protocol informatics), but certainly others (e.g. GA’s with coverage in EFS) provide value. At least, that’s what I got from Chalie Miller’s Real World Fuzzing presentation at Toorcon 9. There are certainly many limitations to the EFS approach, but these are well-known. Commercial fuzzers such as Codenomicon and MuSecurity borrow from these GA/cov and PI heuristics, respectively. Of course, BreakingPoint Systems and beSTORM have their own innovations, mostly around intelligent fault detection instead of heuristics and coverage.

Jared DeMott, who wrote EFS, is working on benchmarking (as stated in his BH-US-07 talk) as well as a book with Ari Takanen and Charles Miller (due in 2008). Gadi Evron and Noam Rathaus were also supposed to release a book on “Open Source Fuzzing Tools” that hasn’t made it out yet. Other benchmarking work is also being performed by the open reverse benchmarking project, which was started by Tom Stracener.

The next iteration of these types of tools are clearly going to include informal testing methods, although the degree to where we go with formal methods is yet unknown. From reading recent literature on ATP, I remain skeptical about formal methods. However, one cannot argue with the results of Coverity (i.e model-checking) and Fortify (or Ounce, GrammaTech, or Armorize) in the static analysis space. This is why I recommend both approaches: model-checking or static analysis to handle Type I errors (i.e. false positives), and fuzz/fault-injection testing to handle Type II errors (i.e. false-negatives), both backed by manual verification processes. Results can then be used to generate reusable unit tests that continuously prevent security-related defects.

I would like to see more formal/semi-formal/informal methods in secure design review tools and processes. This area is open for the most amount of research today, and could provide huge wins. The problem is terminology and approaches are widely varied, MITRE with CAPEC, Microsoft has their STRIDE model and TAM tool (and now a TAM-E tool for Enterprises), Octotrike has the Trike threat-modeling methodology (and Octotrike tool) and the Privilege-Centric Security Analysis, CMU has OCTAVE, and BSI/DHS/Cigital has Architectural Risk Analysis. Then there are several other methods that are derived from rating/scoring systems such as Microsoft DREAD, FIRST’s CVSS[2], Wysopal’s work on combining CWE with CVSS2, and HP/SPI-Dynamics’ Web Application Risk Modeling (W.A.R.M.). If further research is done in this area to turn it into more of science, then this can be easily passed down to security testing by designing/enhancing the test cases.

I should have described myself more clearly when differencing application vs. network penetration testing. The difference I see is that in an network pen test most of the devices and programs you’ll find running are well known, most of these have previously been audited and there might be known vulnerabilities for some versions of these; on the other hand, custom applications require a different approach (I’m not saying that all applications are custom, but you naming fuzzers and application scanners seems to agree with this). In a network penetration test the focus is in assessing what could an attacker do (with the publicly known information, e.g., known vulnerabilities) and is typically restricted to exploits available to the pen tester; say, beacause developing an exploit for a binary vulnerability is much more difficult (these days) than finding an injection vulnerability and developing the exploit for it. So often, in a pen-test with a fixed duration a black box approach is the only possibility.

We have been doing research in pen testing for several years at Core Security Technologies. In their PacSec talk Ivan and Gera show several facts (http://pacsec.jp/psj03/en/2-3iarce-gera-PacSec-JP2003.ppt), say, they provided a methodology to analyze computer attacks and show that some “paths” are more efficient than others. This work stands on the model of Futoransky et al. (http://www.coresecurity.com/files/attachments/Futoransky_Notarfrancesco_Richarte_Sarraute_NetworkAttacks_2003.pdf)
Further, Tiscornia and Russ recently presented a generalization of the above model to applications and a very interesting implementation of some of their ideas in Hack.lu and PacSec. We continue this line of research as we think there is a lot to learn about penetration testing.

I must say that web application scanners have proved useful in certain situations. Yet black, grey and white analysis still throw loads of false positives and false negatives and require many hours to analyze a single application. At present, automated detection of vulnerabilities cannot replace manual work. Hence, the systematic analysis of these tools will help to improve them. But we need to make it right. You cite some of the first attempts on this in your answer to Acidus. Some of these are closely related tothe work in software analysis; and we needn’t reinvent the wheel (e.g., the many known problems with coverage and heuristics that attack these problems). Personally, I still do not see the relevance of some of these measures. I am convinced that user intervention is necessary and therefore prefer to concentrate on helping these users with their work (e.g., replay information for all alarms to eliminate false positives or even automated exploit construction when possible).

Two last bits:
-Going back to the “art” argument and considering the dozens of available tools that are used for assessing the security of a web application I think that there is little art in them (and the underlying pen tests).
-What I meant by the cryto desings I mentioned is that there is no security proof accompaying them. They are only believed to be secure because of their untarnished reputation (an maybe some sound but insufficient arguments).

Cheers

Actually, application pen-testing and network pen-testing are the same thing. Software security penetration-testing could be combined with source-code to provide a hybrid approach directly, by using the information from source-code to improve the test cases for the software security penetration-testing.

Application penetration-testing could also utilize source code, in a way similar to how W3AF plans to use path-traversal or other predictable locations to identify the source and then use it for additional test cases. Since network pen-testing includes application pen-testing, it could use it in the same way. Similarly, application and network pen-testing do not have to be zero-knowledge, and could incorporate aspects of source-code and/or configurations, other customizations, and environmental factors (e.g. architectural threat-modeling) to improve test cases.

I understand the goal of pen-testing in an assessment process, but I am questioning the validity and usefulness of black-box assessments.

The crypto designs you mentioned were not designed as a art, but as a science. Most modern applications are not designed like crypto systems. With crypto systems, these are usually designed and tested using formal, semi-formal, or informal methods. Pen-testing (included manual or automatic, using fuzz testing or fault-injection - as well as other techniques) is neither formal, semi-formal, or informal. It is “ad-hoc” and therefore an art because it has no predefined logical flow, no scientific input or output, and precludes the use of both proof or arguments for correctness. I will get to the differences in more detail in a later blog post.

If you can point me towards systematic research that includes at least “arguments for correctness” so that at least points in logical errors of testing can at least be described, then I will slightly concede your point. I’m working on this type of research with a team of individuals but it has just started and we haven’t released anything yet. I’m not at liberty to discuss it yet (another future blog post), but I can tell you that it has to do with benchmarking automated fault-injection scanners.

We can measure the coverage of a software or application penetration-test very easily using code coverage tools, however this does not provide a benchmark for the tool, as it doesn’t take into account false positives or false negatives, nor does it take into account configuration/customization or the tester behind the testing tool. Take a look of what I posted in response to Acidus on memestreams. I don’t want to say that this will be part of a later blog post, but hey - that’s one of my best answers right now because of my limited time in responding to your post here. I will try to respond to further questions if you have any more.