Comments on: Scraping the web for fun and profit http://www.tssci-security.com/archives/2007/10/18/scraping-the-web-for-fun-and-profit/ top secret/secure computing information Sat, 05 Jul 2008 20:30:40 +0000 http://wordpress.org/?v=2.2.3 By: Marcin http://www.tssci-security.com/archives/2007/10/18/scraping-the-web-for-fun-and-profit/#comment-2281 Marcin Wed, 07 Nov 2007 16:15:51 +0000 http://www.tssci-security.com/archives/2007/10/18/scraping-the-web-for-fun-and-profit/#comment-2281 This was posted by dre, not me. We have a couple people blogging on our team :) * <strong>Marcin</strong>, (me) * <strong>Casey</strong>, OS X guru and * <strong>dre</strong>, an everything guru This was posted by dre, not me. We have a couple people blogging on our team :)

* Marcin, (me)
* Casey, OS X guru and
* dre, an everything guru

]]> By: Travis http://www.tssci-security.com/archives/2007/10/18/scraping-the-web-for-fun-and-profit/#comment-2276 Travis Wed, 07 Nov 2007 13:20:25 +0000 http://www.tssci-security.com/archives/2007/10/18/scraping-the-web-for-fun-and-profit/#comment-2276 nice article, i am reading webbots, spiders, and screen scrapers as well. although it may not be the best way to scour the web its certainly an interesting read. languages like python, ruby, and perl may or may not be more powerful than the php curl combination. either way Marcin i like your mention of all the available parsing options and of course google alerts. http://travisaltman.com nice article,

i am reading webbots, spiders, and screen scrapers as well. although it may not be the best way to scour the web its certainly an interesting read. languages like python, ruby, and perl may or may not be more powerful than the php curl combination. either way Marcin i like your mention of all the available parsing options and of course google alerts.

http://travisaltman.com

]]> By: Casey http://www.tssci-security.com/archives/2007/10/18/scraping-the-web-for-fun-and-profit/#comment-2058 Casey Tue, 23 Oct 2007 20:18:16 +0000 http://www.tssci-security.com/archives/2007/10/18/scraping-the-web-for-fun-and-profit/#comment-2058 Simplicity and it's faster from what I have seen. This will grab all links on a page that are in list items inside of an ordered list. So simple and easy to write. require 'rubygems' require 'scrapi' links = Scraper.define do process "ol>li>a[href]", "urls[]"=>"@href" result :urls end puts links.scrape(URI.parse("http://ezinearticles.com/?cat=Business")) Simplicity and it’s faster from what I have seen. This will grab all links on a page that are in list items inside of an ordered list. So simple and easy to write.

require ‘rubygems’
require ’scrapi’

links = Scraper.define do
process “ol>li>a[href]”, “urls[]”=>”@href”
result :urls
end

puts links.scrape(URI.parse(”http://ezinearticles.com/?cat=Business”))

]]> By: dre http://www.tssci-security.com/archives/2007/10/18/scraping-the-web-for-fun-and-profit/#comment-2056 dre Tue, 23 Oct 2007 19:00:59 +0000 http://www.tssci-security.com/archives/2007/10/18/scraping-the-web-for-fun-and-profit/#comment-2056 Why would you rather use CSS locators than XPath? I've heard of people wanting to use the DOM selectors - for example in the case of Selenium tests being run under Internet Explorer (apparently the XPath ones are very slow), but never CSS locators. html > body > div #wrap > div #content > div .post > div #commentsection > form #commentform > p > textarea #comment or id('comment') right? Why would you rather use CSS locators than XPath? I’ve heard of people wanting to use the DOM selectors - for example in the case of Selenium tests being run under Internet Explorer (apparently the XPath ones are very slow), but never CSS locators.

html > body > div #wrap > div #content > div .post > div #commentsection > form #commentform > p > textarea #comment

or

id(’comment’)

right?

]]> By: Casey http://www.tssci-security.com/archives/2007/10/18/scraping-the-web-for-fun-and-profit/#comment-2042 Casey Tue, 23 Oct 2007 00:45:46 +0000 http://www.tssci-security.com/archives/2007/10/18/scraping-the-web-for-fun-and-profit/#comment-2042 I use scrubyt and scrapi for scraping. I prefer scrapi though because I'd rather use CSS selector id's rather than xpaths. I use scrubyt and scrapi for scraping. I prefer scrapi though because I’d rather use CSS selector id’s rather than xpaths.

]]>