Web application security incident handling
I thought I’d take a moment to post about some web security tools I use pretty often, which help as a security consultant when responding to various web hacking related incidents. These tools have helped me write my own scripts whenever I’m in a jam and need something good and quick to do the job.
Application Log File Forensics: The Hard Way
The first thing a security professional or administrator usually think of when handling an application security incident is to check the logs for the applications, databases, and other application-tiers involved. Often, these logs are either on the servers that run the applications themselves, or possibly in a central logging location. If a certain attacker tool can be identified from the log files (or other sources such as full packet-capture), then it may be of interest to run that exact same tool against your own application-under-target (preferably in a mocked-up lab or test environment, if it mirrors production well enough).
The most popular web servers, Apache httpd and Microsoft IIS, do create local log files by default. According to most compliance regulations and standards (e.g. COBIT, HIPAA, GLB, PCI-DSS, FISMA, EU Directive on Privacy and Electronic Communications, ISO 17799/27002, CA SB1386 and similar), logging must be centrally located, or may have other required provisions. This may include application-layer information, such as the log information from Apache and IIS. It may be very likely that your organization already has centralized logging where this information is available.
If centralized logging does not exist, it may be a good time to start up a project to enable it. The Apache Cookbook, 2E, is the best place to go in order to configure httpd to start sending syslog information. It’s about as simple to add “ErrorLog syslog:user” into the httpd.conf file, but this only logs error messages, not authentication/access_log messages. The book gives two prescriptions, one using “AccessLog “|/usr/bin/logger” combined” if your OS supports the logger command properly. The other is to run a custom log message through a Perl script, as seen below:
CustomLog |/usr/local/apache/bin/apache_syslog combined
cat > apache_syslog
#!/usr/bin/perl
use Sys::Syslog qw( :DEFAULT setlogsock );
setlogsock('unix');
openlog('apache', 'cons', 'pid', 'user');
while ($log = <STDIN>) {
syslog('notice', $log);
}
closelog;
Microsoft IIS will need to go through the Event Log, which can be converted to syslog messages using a third-party software package such as Snare or MonitorWare Agent. If IIS logs can also be converted to w3c standard log format, then Apache log analyzer tools such as AWStats could also be used. W3C also has their own log analysis tool that also does HTML validation, called the Log Validator. These may be useful to run following your own scan of the application using the same or similar attacker tool, as they will not only point out where in your application the scan/tool covered, but also where you may have the most errors or lack of quality/security controls.
The book Practical Information Security Monitoring also makes some suggestions for log collections, including the use of Sawmill or Splunk to sort/search log messages and gain further information and detail. There may also be further adjustments you will want to do at the application (or other tier) layer, such as logging POST data. We discussed logging HTTP referrers on our old post: Using Google Analytics to Subvert Privacy. Practical Information Security Monitoring talks about Oracle audit logging, but there is also a detailed article on Pete Finnigan’s blog on Oracle forensics and UKOUG. At the recent BlackHat DC conference, David Litchfield gave a talk on The Forensic Investigation of a Compromised Oracle Database Server, which may also be of interest (once the slides are available). There are also some new books coming out on the topic of Oracle Forensics in the next few months / year.
Web Application Incident Handling: The Easy Way
Most of the logfile “digging” takes time, even when consolidated and using expert tools and analysis. There are some very easy approaches that we’ve come up with, or seen others using and talking about. These tools integrate well at the HTML and Script layers.
Over a year ago, Mario Heiderich started the PHP-IDS project, as a way to build protection and monitoring capabilities into PHP applications. Several side projects spurred up as a direct result of the incredible work that was put into PHP-IDS, mainly its default_filter.xml regular expressions. This XML file of regular expressions provides capabilities to detect a vast range of attacks, including XSS, CSRF, SQL Injection, Directory Traversal, Local/Remote File Execution, DoS, and Information Disclosure. Part of the success behind the PHP-IDS project, was the constant testing and attacking of PHP-IDS regex filters, which can be reviewed extensively in this sla.ckers.org thread. More info on PHP-IDS can be found in the PHP-IDS FAQ.
Romain Gaucher, wrote Scalp, an Apache log analyzer in Python, which leverages PHP-IDS’ default_filter.xml to detect attack strings in logs. I’ve used Scalp on numerous occasions, including a recent attack attempt on tssci-security.com. By nature, Scalp cannot examine POST content because Apache logs do not contain POST data. (See PHP-IDS or mod_security for those purposes)
Simply use Scalp by running it as follows (keep in mind there may be false positives with regards to the attack type, though it is very good at pulling attack queries from the log):
./scalp.py --log access_log --filters ./default_filter.xml --html --tough --exhaustive
Arshan Dabirsiaghi recently released OWASP Scrubbr. Scrubbr works by detecting input data in a specified database that does not match up with a specified AntiSamy policy file. Because Scrubbr uses an AntiSamy policy to validate data, does not mean it necessarily detects XSS in your database. Note, one does not require AntiSamy to be implemented in an application to use Scrubbr. Using Scrubbr, you have the capability of validating each and every column capable of holding strings of every row of every table in a database.
Together, Scalp and Scrubbr make for excellent web application security forensic tools. Scalp can help detect attacks in Apache logs, and Scrubbr can help you clean your database of content that does not match your site’s policy.
There is an interesting article from the Google Webmaster Central blog on Best Practices Against Hacking. In the post, there are some rough guidelines for application developers, but what I found interesting were the sections on Some ways to identify the hacking of your site and Hacked behavior removed, now what?.
Google suggests setting up Google Alerts like a clipping service to detect malware or blog spam on your own web site or blog. They also show how malware can be found using the Google Webmaster Tools, as well as how to reconcile your Google malware status if your site has been used for spam or other nefarious purposes.
Also of interest are a few other books. SQL Hacks has a hack on Processing Web Server Logs that discusses some interesting ways of querying SQL databases that have imported web server (Apache, IIS) logs. Practical mod_perl speaks to Server Maintenance Chores, which discusses the many performance and other maintenance issues regarding web server log files, including some hints to deal with DDoS and spam targeted at the HTTP layer. Building Scalable Web Sites has a section on Statistics, Monitoring, and Alerting that goes into detail about customization of environmental variables using mod_rewrite into the Apache log format interface. Sams Teach Yourself Apache 2 in 24 Hours has an hour dedicated to Logging and Monitoring with one of the best examples of Apache’s log format directives and information on the Common Log Format (CLF). My favorite documentation on Apache’s log format directives is from Ivan Ristic’s Apache Security book, which is mandatory reading for anyone implementing Apache httpd — the entire chapter on Logging and Monitoring goes way more in-depth than any other title on the subject matter I’ve seen yet. For IIS, you’ll want to check out the two Syngress press titles: Microsoft Log Parser Toolkit and Security Log Management, which cover the IISW3C Input Format in great detail and Log File Conversion topics, respectively. Another book for IIS administrators is the Microsoft Windows Server 2003 Insider Solutions: Shortcuts and Best Practices, which covers Monitoring IIS Access Through Auditing and Logging, although the Microsoft Press titles on your specific version of IIS will certainly be worthwhile reads, as well.
My favorite overall read by far has been the most recent SQL Server Forensic Analysis by Kevvie Fowler! Kevvie also spoke about SQL Server Forensics at the most recent BlackHat US with his presentation on [PDF] SQL Server Database Forensics and the co-existing whitepaper on [PDF] A Real World Scenario of a SQL Server 2005 Database Forensics. These blew my mind away. In the book, Kevvie Fowler shows how to trace back SQL injection from IIS logs into the “plan cache” of a SQL Server 2005 instance. Using this information, an incident handler can easily figure out not only that data has been or could have been breached, but which specific data and in what way. The book is highly technical, so I might suggest that people just introducing themselves with Windows Forensics start with Mastering: Windows Network Forensics and Investigation, another good book that covers IIS logs, Microsoft Log Parser, the Event Viewer, and audit events.
Considering the amount of recent breachery via SQL injection in high profile websites, I think that the web application tier is going to need some major improvements to incident handling approaches. It’s not like SQL injection has been making news much lately. I can’t seem to find any examples of SQL injection in the news, right?