People are going to continue to call both pen-testing, so I suggest that you and I both get used to it.”

-agreed, i’ll still my best do call if software assurance here so i dont get confused ;-)

“Your laptop strategy is interesting, although what about boot rootkits?”

-I suppose the threat is there, but no more of a threat than the switch i just plugged into already being owned as well, straight from China.

“As for the data liability, I still think it’s a problem even if you don’t find PII such as health information.”

-Its an interesting point, I guess you mean from a theft after its retained point of view. Something to consider to say the least. have you heard of it happening in the past? information assurance at the network level has been going on for years. any cases of those people taking liberties with that data. For the most part, the data we are asked to see if we get to we have a vested interest in safeguarding as well AND has little value on “the market”.

Please never use the word, “audit” to describe the process of breaking software. That’s applying an operational/compliance “C” word to something very technical and tactical.

The only time “software” and “audit” will ever go together will be with PA-DSS, which is bound to be as much as an abomination as PCI-DSS.

I wish everyone would use the terms
1) Information assurance to describe what you do
2) Software assurance to describe what I do

People are going to continue to call both pen-testing, so I suggest that you and I both get used to it.

Your laptop strategy is interesting, although what about boot rootkits? As for the data liability, I still think it’s a problem even if you don’t find PII such as health information. All exploits are dangerous weapons of mass destruction. Play with them in a lab and not on the battlefield.

-I’m sure that wasnt a you=me, specifically Chris, but a pentesters that do network auditing. as a far as a liability, you mean like someone on the team does something with that data? The data we look for is operational and not credit or health. In the event we do come across that, and we shouldnt, people are trained on how to properly handle and safeguard that. Something like that generally means a stop work and get it fixed immediately.

“Also, I’m not calling it pen-testing. That’s Matasano’s definition of the word that I’m using. In this case, yes, I’m talking about strictly software pen-testing. Would it help if I used the words `vulnerability assessment’ instead?”

-not to argue definitions, but you use pentesting in your posts alot, but I guess calling it a software audit would be better (for me anyway), unless there has been a fundamental definition change. sounds like a good chance to coin a new term, i dont think pentesting fits what you are trying to do though. dont get me wrong, i think there is tons of value in making sure the software is secure long before it gets on a network.

“With specific regard to network pen-testing, I think this is a messy situation for many companies. I appreciate that people such as yourself can go in, analyze, and provide reports. How do you remediate?”

-remediate, you should know as well as anyone else its advice time at that point. we have no control over those networks, i cant MAKE anyone make any changes afterwards. We always have a tech outbrief before we leave to talk to the admins so they can start working fixes between the time when we leave and they get the report, this is probably not the norm but something we do based on the customers.

“I’m actually most worried about someone like you getting owned before, during, or after a pen-test with all of the data/exploits you go through for clients. How do you handle this problem?”

-Are you really? doubtful. software is vetted, computers used for the audit only, data is burned to CD, then hard drives purged and reimaged for the next assessment.

“I’ve heard of some vulnerability assessment / pen-testers / auditors that run only a few simple read-only programs while working with a client…”

-network pentester our software pentesters/auditors? if its network guys i’d appreciate knowing who so i can try to pick their brain.

i think i got the critical questions, let me know if i missed one.

-CG

See, I guess where we start to disagree is where I think that network pen-testing (such as what you describe) goes wrong. In many ways, I think it’s unethical to do what you do.

It’s ok that we disagree about this. I find it surprising that people will hire you to actually break into real systems and steal real data. Isn’t that a liability?

If we can test COTS applications/devices in a lab, then why wouldn’t we want to do it there?

Also, I’m not calling it pen-testing. That’s Matasano’s definition of the word that I’m using. In this case, yes, I’m talking about strictly software pen-testing. Would it help if I used the words `vulnerability assessment’ instead?

With specific regard to network pen-testing, I think this is a messy situation for many companies. I appreciate that people such as yourself can go in, analyze, and provide reports. How do you remediate?

I’m actually most worried about someone like you getting owned before, during, or after a pen-test with all of the data/exploits you go through for clients. How do you handle this problem? In my case, I want read access to the source code — but I don’t want write access to the revision control repository. Is there an equivalent for network pen-testing?

Companies need to focus on the non-technical aspects in order to solve the types of problems that network pen-testing finds. This doesn’t necessarily involve rigid change control, but it does involve some kind of change management with tracking. It mostly involves making sure that all machines/applications are fully-patched — and for systems/apps that are not fully-patched, some sort of compensating control in place.

Today, a popular compensating control is network-based IPS. Many people still love network firewalls and network-based AV as well. Then you get to the host-based IPS, AV, and software firewalls. However, for every measure (i.e. protection) here is often another counter-measure (i.e. evasion). It becomes a game of cat-and-mouse.

When dealing with moving from a state of the unknown to a state of “we have a large majority, if not all, known vulnerabilities here in this report”, I think that rolling out a patch management system is about as good as any other approach. This way you get the one-offs that might not be connected to the network during the pen-test. I like the OVAL-Compatible products because it seems to me that this is a good way to share data across products. I’m also a fan of passive notification of vulnerability information, such as that found via Cassandra, Advisory check, SIGVI, and found in the OSVDB 2.0 database.

The problem is that even patch management solutions are another compensating control in a way, and these products are often vulnerable in many ways themselves. Patch management, anti-virus, firewalls, intrusion protection, and scanning/exploitation-engines are all some of the weakest software (with the chewiest centers) out there. This is why I recommend that organizations uninstall host-based AV/IPS once they have verified that their systems are fully patched (and have a verified procedure in place to keep them patched). Defense-in-depth is sometimes the worst strategy to employ.

I’ve heard of some vulnerability assessment / pen-testers / auditors that run only a few simple read-only programs while working with a client. They download the source code, find the bugs, make their reports, and get out. They don’t use nmap or Nessus — let alone Core Impact RPT. They don’t even use a web browser (or have one installed) unless it’s w3m or a custom curl script. They have personally verified every piece of software on their own computer by manually examining the source code to these open-source applications for themselves. When they use the network or send files, they do so using secure channels and encryption.

On a pentest, where I come in to do a network level audit, looking for where you didn’t patch machines, flaws in your network design, then if I can attack your clients, attacking them and your defenses to that, most of what you wrote wont work.

While time management is of course an issue, some spiffy calendar ringing at 1515 saying now I should be scanning Y instead of X isn’t very realistic. A far better solution is a solid methodology to get you started on your assessment, a way of doing business every time on every assessment.

MITRE is great I really like the Common Attack Pattern Enumeration and Classification (CAPEC), and I want to spend some more time with it, but most customers want demonstratable vulnerabilities…meaning I need to either get data or get a shell from the vulnerability. Listing 2000 vulnerabilities with your web app or unpatched vulnerabilities that don’t have exploit code are a hard sell. Would I list is that stuff yes, but I’m not here to do a nessus scan, the local admin can do that. We’re usually there to take an outsider’s look at your security in place. At some point that means I need to actually break into something or steal data.

For 3, most stuff I see is COTS or built by someone else and the admins are just stuck running it. its far too late for SDLC or code reviews now. I do realize that I don’t do the same type of testing that everyone else does.

4 is the similar to 3 but if my customer cant touch the code, what can they do then? 5-7 are great.

Our reality is shaped by the world we live and WORK in, so what I see I realize isn’t what EVERYONE else sees, but I think for most “come audit my network” type pentesters we are way far past the SDLC, all the things you listed are great if I can get ahold of that app or network before its rolled out, but not after its been deployed.

You said, “Though, what if pen-tester is doing pen-testing without the involvement of Company staff? OR your steps are just relevant to web applications? not overall pentesting.”

Without involvement of company staff? You make that sound illegal. I think that I do know what you mean — you’re trying to say that it’s just a manager who works with internal auditors or similar?

In this case, I suggest strategy consulting, especially one of the greatest tenants of strategy consulting: informational interviews.

Also, as I discussed above, I was talking about both web applications and fat applications.

Great piece of work. You really have awesome knowledge. After reading your post i do learn something out of it. Keep up the good work.

The steps you have mentioned to be a good pen-tester is really good. Though, what if pen-tester is doing pen-testing without the involvement of Company staff? OR your steps are just relevant to web applications? not overall pentesting….

I like the post by Matasano Chargen…

http://www.matasano.com/log/719/more-on-pen-testing-2/

I would really emphasize on step one… i guess that where alot of pen-testers are getting failed.

Hackers doesn’t have any SLA, pen-testers do have one.

Cheers

Shoaib