BLU Discuss list archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Discuss] What's the best site-crawler utility?

Subject: [Discuss] What's the best site-crawler utility?
From: greg at freephile.com (Greg Rundlett (freephile))
Date: Tue, 7 Jan 2014 23:02:15 -0500
In-reply-to: <52CCC4C8.70901@gmail.com>
References: <52CC9277.4010309@horne.net> <52CC9B92.4030104@mattgillen.net> <52CCC4C8.70901@gmail.com>

Also, I just discovered a MediaWiki extension written by Tim Starling that
may suit your needs.  As the name implies, its for dumping to HTML.

http://www.mediawiki.org/wiki/Extension:DumpHTML

As for processing the XML produced by "export" or MediaWiki dump tools,
here is info on that XML schema
http://meta.wikimedia.org/wiki/Help:Export#Export_format

And, some of the tools you can use to process MediaWiki XML
http://wikipapers.referata.com/wiki/List_of_data_processing_tools


Greg Rundlett

References:
- [Discuss] What's the best site-crawler utility?
  - From: bill at horne.net (Bill Horne)
- [Discuss] What's the best site-crawler utility?
  - From: me at mattgillen.net (Matthew Gillen)
- [Discuss] What's the best site-crawler utility?
  - From: tmetro+blu at gmail.com (Tom Metro)

Prev by Date: [Discuss] What's the best site-crawler utility?
Next by Date: [Discuss] Small website, non-technical users: Joomla, Drupal, or WordPress? (Solved)
Previous by thread: [Discuss] What's the best site-crawler utility?
Next by thread: [Discuss] What's the best site-crawler utility?
Index(es):
- Date
- Thread


BLU is a member of BostonUserGroups
We also thank MIT for the use of their facilities.

Boston Linux & Unix / webmaster@blu.org