Qualities of good pen-testers
Taking care of business
Before I get into this post, I wanted to give you some updates on progress of other projects here at TS/SCI Security.
First off, I’ve been working on the OWASP Evaluation and Certification Criteria Project and hope to announce something very soon. Secondly, you’ll want to take a look at today’s post on The New School of Information Security book (Acidus also did a writeup). As a side-note, there was an interesting data breach recently announced, which makes information on breaches even that much more relevant.
Lastly, I wanted to announce the return of the “Day X: ITSM Vulnerability Assessment techniques“. Expect a post everyday with new and relevant material.
Positive spin on pen-testers
Matasano posted last week on the Seven Deadly Pen-Test Sins, with follow-ups from two other blogs I’m aware of: Mike Andrews and BlogFranz. This should be a hot-topic!
At TS/SCI Security, we like to focus on the positive instead of the negative. Pen-testing, especially application pen-testing/review is an important topic to us. So here’s our rundown of what we feel is important to know about pen-testing, as a response to what Matasano wrote.
Seven things you can do to improve your pen-test results
1. Time Management. If you’re not already using GCalendar, RTM (with GCal), Twitter, Sandy, and Jott (thanks to Dennis Groves for some of these!) — take a look at how these can help you manage your time better.
- Good pen-testing takes time. If you haven’t gone through everything you need in #2, then you might need more time. What I’ve found is that pen-testers need to frame their pen-tests after good strategy consulting (1-2 days of a SWOT or similar analysis). If, like mentioned in The New School, the “basics” are not taken care of — then help the client work on asset, configuration, and change management before doing a vulnerability assessment.
2. Utilize applicable MITRE resources. CAPEC (for runtime testing) and CWE (for static analysis testing, which may include reversing) should be utilized throughout. Be careful to only choose the relevant aspects of each (relevant to the application, not your skill or other criteria).
- My favorite MITRE resource is the Introduction to Vulnerability Theory, which Marcin and I spoke about at ShmooCon in our talk on Path X. It may take some extra time to spell out exactly how the vulnerabilities were found using this method, but it will help future assessment work, including repeat client business.
3. Work with developers. Assuming that the dev team is already doing Continuous Integration (or something like it), Fagan inspection, using the V-Model (in the requirements phase), continuous-prevention development, integrating concepts from the Microsoft SDL (or similar secure SDLC), and automated integration testing — then get involved right away with a software walkthrough with all of the key players.
- Standard C and assembly code is difficult to reverse the design from, but they are becoming more easy to reverse. In the case of Java, C#, and C++ — UML can be extracted to elicit the design of the application. Using other tools such as Klocwork K7, Fujaba, or IBM Rational Rose on the UML diagrams will possibly provide faster program understanding. Left with the source code, GrammaTech CodeSurfer also aids understanding, but open-source tools such as Doxygen can also be of use.
4. Automate development-testing. While web application security scanners (or fat application-based fuzzers) collectively only typically find at-most 30% of the bugs with a large amount of false positives, there are plenty of developer tools that don’t have these problems.
- Developers should be made aware of integration testing with Canoo WebTest. Business analysts, customers, and test managers can submit table-driven test cases using FitNesse (a wiki that allows for collaborative test case design), along with additional tools such as HtmlFixture. I’ve spoke to the benefits of Dependency Injection before, as well as automating Continuous Integration both inside of the IDE as well as during the application builds.
- Automation is good for developers and the modern tools surrounding test automation are extreme improvements over last decade’s technology. However, test cases and test-first strategies still find only as much as half (possibly up to 70% or more) of the bugs. It’s good to combine exploratory testing with scripted (i.e. from a test case) testing. Allot some time with a buddy and explore the application as it wasn’t meant to be run after learning as much as you can from the developers and their automated tests.
5. Peer review your work. This one is easy. Find a friend and have him/her check your work. The more eyes you get on your pen-test and assessment work, the less likely you’re going to miss something.
- Consider a good workflow approach such as the open-source Review Board: Online Code Review Tool or the commercial Atlassian set of project management software, including JIRA, FishEye, and Crucicble. Many projects such as Adium do code review based on email notification of new source code revision control changes.
6. Stay up-to-date on both process and technology. Read forums/blogs/mailing-lists, trade journals/magazines, and books. Attend conferences. Keep it cheap and continuous if possible.
- I try to read every security-related book that pops up on Safari Library and ACM’s access to Books24×7. Audible/Amazon are certainly other resources, especially for non-technical solution learning. Checking yourself at least once a year on a certification or re-certification program will make strides in your professional development.
- Using GReader and sharing daily using it’s sharing features and OPML with your colleagues is important. Get involved in conversations on forums, IRC channels, blogs, and mailing-lists is likely the largest win here.
7. Go back to basics. Set a minimum bar of maturity with clients before you pen-test or do an assessment. The BITS Shared Assessments Standardized Information Gathering questionnaire is a nice start, especially combined with Visible Ops, ITIL, COBIT, ISO27K, PCI-DSS, PABP/PA-DSS, ISECOM, and OWASP information on process.
All of the above is going to go a long way towards improvements to what pen-testers and vulnerability assessors do. There’s very few good certifications out there for what we do, and it’s difficult to measure what we do. Put the cards on the table up front with your clients and teach them your methodology and approach.
Dre,
Great piece of work. You really have awesome knowledge. After reading your post i do learn something out of it. Keep up the good work.
The steps you have mentioned to be a good pen-tester is really good. Though, what if pen-tester is doing pen-testing without the involvement of Company staff? OR your steps are just relevant to web applications? not overall pentesting….
I like the post by Matasano Chargen…
http://www.matasano.com/log/719/more-on-pen-testing-2/
I would really emphasize on step one… i guess that where alot of pen-testers are getting failed.
Hackers doesn’t have any SLA, pen-testers do have one.
Cheers
Shoaib
@ Shoaib:
You said, “Though, what if pen-tester is doing pen-testing without the involvement of Company staff? OR your steps are just relevant to web applications? not overall pentesting.”
Without involvement of company staff? You make that sound illegal. I think that I do know what you mean — you’re trying to say that it’s just a manager who works with internal auditors or similar?
In this case, I suggest strategy consulting, especially one of the greatest tenants of strategy consulting: informational interviews.
Also, as I discussed above, I was talking about both web applications and fat applications.
I’m going to start by disagreeing with what you call pen-testing in your post. Maybe closer to application testing is what you propose, in that case the things you wrote might work well for someone who is called in to audit a new application built by the customer or audit a web application but not their WHOLE network.
On a pentest, where I come in to do a network level audit, looking for where you didn’t patch machines, flaws in your network design, then if I can attack your clients, attacking them and your defenses to that, most of what you wrote wont work.
While time management is of course an issue, some spiffy calendar ringing at 1515 saying now I should be scanning Y instead of X isn’t very realistic. A far better solution is a solid methodology to get you started on your assessment, a way of doing business every time on every assessment.
MITRE is great I really like the Common Attack Pattern Enumeration and Classification (CAPEC), and I want to spend some more time with it, but most customers want demonstratable vulnerabilities…meaning I need to either get data or get a shell from the vulnerability. Listing 2000 vulnerabilities with your web app or unpatched vulnerabilities that don’t have exploit code are a hard sell. Would I list is that stuff yes, but I’m not here to do a nessus scan, the local admin can do that. We’re usually there to take an outsider’s look at your security in place. At some point that means I need to actually break into something or steal data.
For 3, most stuff I see is COTS or built by someone else and the admins are just stuck running it. its far too late for SDLC or code reviews now. I do realize that I don’t do the same type of testing that everyone else does.
4 is the similar to 3 but if my customer cant touch the code, what can they do then? 5-7 are great.
Our reality is shaped by the world we live and WORK in, so what I see I realize isn’t what EVERYONE else sees, but I think for most “come audit my network” type pentesters we are way far past the SDLC, all the things you listed are great if I can get ahold of that app or network before its rolled out, but not after its been deployed.
@ CG:
See, I guess where we start to disagree is where I think that network pen-testing (such as what you describe) goes wrong. In many ways, I think it’s unethical to do what you do.
It’s ok that we disagree about this. I find it surprising that people will hire you to actually break into real systems and steal real data. Isn’t that a liability?
If we can test COTS applications/devices in a lab, then why wouldn’t we want to do it there?
Also, I’m not calling it pen-testing. That’s Matasano’s definition of the word that I’m using. In this case, yes, I’m talking about strictly software pen-testing. Would it help if I used the words `vulnerability assessment’ instead?
With specific regard to network pen-testing, I think this is a messy situation for many companies. I appreciate that people such as yourself can go in, analyze, and provide reports. How do you remediate?
I’m actually most worried about someone like you getting owned before, during, or after a pen-test with all of the data/exploits you go through for clients. How do you handle this problem? In my case, I want read access to the source code — but I don’t want write access to the revision control repository. Is there an equivalent for network pen-testing?
Companies need to focus on the non-technical aspects in order to solve the types of problems that network pen-testing finds. This doesn’t necessarily involve rigid change control, but it does involve some kind of change management with tracking. It mostly involves making sure that all machines/applications are fully-patched — and for systems/apps that are not fully-patched, some sort of compensating control in place.
Today, a popular compensating control is network-based IPS. Many people still love network firewalls and network-based AV as well. Then you get to the host-based IPS, AV, and software firewalls. However, for every measure (i.e. protection) here is often another counter-measure (i.e. evasion). It becomes a game of cat-and-mouse.
When dealing with moving from a state of the unknown to a state of “we have a large majority, if not all, known vulnerabilities here in this report”, I think that rolling out a patch management system is about as good as any other approach. This way you get the one-offs that might not be connected to the network during the pen-test. I like the OVAL-Compatible products because it seems to me that this is a good way to share data across products. I’m also a fan of passive notification of vulnerability information, such as that found via Cassandra, Advisory check, SIGVI, and found in the OSVDB 2.0 database.
The problem is that even patch management solutions are another compensating control in a way, and these products are often vulnerable in many ways themselves. Patch management, anti-virus, firewalls, intrusion protection, and scanning/exploitation-engines are all some of the weakest software (with the chewiest centers) out there. This is why I recommend that organizations uninstall host-based AV/IPS once they have verified that their systems are fully patched (and have a verified procedure in place to keep them patched). Defense-in-depth is sometimes the worst strategy to employ.
I’ve heard of some vulnerability assessment / pen-testers / auditors that run only a few simple read-only programs while working with a client. They download the source code, find the bugs, make their reports, and get out. They don’t use nmap or Nessus — let alone Core Impact RPT. They don’t even use a web browser (or have one installed) unless it’s w3m or a custom curl script. They have personally verified every piece of software on their own computer by manually examining the source code to these open-source applications for themselves. When they use the network or send files, they do so using secure channels and encryption.
“It’s ok that we disagree about this. I find it surprising that people will hire you to actually break into real systems and steal real data. Isn’t that a liability?”
-I’m sure that wasnt a you=me, specifically Chris, but a pentesters that do network auditing. as a far as a liability, you mean like someone on the team does something with that data? The data we look for is operational and not credit or health. In the event we do come across that, and we shouldnt, people are trained on how to properly handle and safeguard that. Something like that generally means a stop work and get it fixed immediately.
“Also, I’m not calling it pen-testing. That’s Matasano’s definition of the word that I’m using. In this case, yes, I’m talking about strictly software pen-testing. Would it help if I used the words `vulnerability assessment’ instead?”
-not to argue definitions, but you use pentesting in your posts alot, but I guess calling it a software audit would be better (for me anyway), unless there has been a fundamental definition change. sounds like a good chance to coin a new term, i dont think pentesting fits what you are trying to do though. dont get me wrong, i think there is tons of value in making sure the software is secure long before it gets on a network.
“With specific regard to network pen-testing, I think this is a messy situation for many companies. I appreciate that people such as yourself can go in, analyze, and provide reports. How do you remediate?”
-remediate, you should know as well as anyone else its advice time at that point. we have no control over those networks, i cant MAKE anyone make any changes afterwards. We always have a tech outbrief before we leave to talk to the admins so they can start working fixes between the time when we leave and they get the report, this is probably not the norm but something we do based on the customers.
“I’m actually most worried about someone like you getting owned before, during, or after a pen-test with all of the data/exploits you go through for clients. How do you handle this problem?”
-Are you really? doubtful. software is vetted, computers used for the audit only, data is burned to CD, then hard drives purged and reimaged for the next assessment.
“I’ve heard of some vulnerability assessment / pen-testers / auditors that run only a few simple read-only programs while working with a client…”
-network pentester our software pentesters/auditors? if its network guys i’d appreciate knowing who so i can try to pick their brain.
i think i got the critical questions, let me know if i missed one.
-CG
@ CG:
Please never use the word, “audit” to describe the process of breaking software. That’s applying an operational/compliance “C” word to something very technical and tactical.
The only time “software” and “audit” will ever go together will be with PA-DSS, which is bound to be as much as an abomination as PCI-DSS.
I wish everyone would use the terms
1) Information assurance to describe what you do
2) Software assurance to describe what I do
People are going to continue to call both pen-testing, so I suggest that you and I both get used to it.
Your laptop strategy is interesting, although what about boot rootkits? As for the data liability, I still think it’s a problem even if you don’t find PII such as health information. All exploits are dangerous weapons of mass destruction. Play with them in a lab and not on the battlefield.
“I wish everyone would use the terms
1) Information assurance to describe what you do
2) Software assurance to describe what I do
People are going to continue to call both pen-testing, so I suggest that you and I both get used to it.”
-agreed, i’ll still my best do call if software assurance here so i dont get confused ;-)
“Your laptop strategy is interesting, although what about boot rootkits?”
-I suppose the threat is there, but no more of a threat than the switch i just plugged into already being owned as well, straight from China.
“As for the data liability, I still think it’s a problem even if you don’t find PII such as health information.”
-Its an interesting point, I guess you mean from a theft after its retained point of view. Something to consider to say the least. have you heard of it happening in the past? information assurance at the network level has been going on for years. any cases of those people taking liberties with that data. For the most part, the data we are asked to see if we get to we have a vested interest in safeguarding as well AND has little value on “the market”.