User-agent: Googlebot Allow: / User-agent: 164.71.1.147,jp.co.fujitsu.t2 164.71.1.149,jp.co.fujitsu.t4 Robot Disallow: / User-agent: 194.20.32.131 Disallow: / User-agent: 198.5.209.201 Disallow: / User-agent: 204.95.207.62 user agent ? Disallow: / User-agent: 205.252.60.71 Disallow: / User-agent: 4anything.com LinkChecker v2.0 Disallow: / User-agent: 64.57.223.40 robot Disallow: / User-agent: AbachoBOT Disallow: / User-agent: About/0.1libwww-perl/5.47 Disallow: / User-agent: acke.dc.luth.se Disallow: / User-agent: Acme.Spider Disallow: / User-agent: AcoiRobot Disallow: / User-agent: Acoon Robot v1.50.001 Disallow: / User-agent: Ad Muncher v4.xx.x Disallow: / User-agent: AESOP_com_SpiderMan Disallow: / User-agent: Ahoy! The Homepage Finder Disallow: / User-agent: Aladin/3.324 Disallow: / User-agent: Alkaline Disallow: / User-agent: AlkalineBOT/1.3 Disallow: / User-agent: AlkalineBOT/1.4 (1.4.0326.0 RTM) Disallow: / User-agent: Allesklar/0.1 libwww-perl/5.46 Disallow: / User-agent: Alligator 1.31 (www.nearsoftware.com) Disallow: / User-agent: AltaVista Intranet V2.0 AVS EVAL search@freeit.com Allow: / User-agent: AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.net Allow: / User-agent: AltaVista Intranet V2.0 evreka.com crawler@evreka.com Allow: / User-agent: Altavista robot Allow: / User-agent: Altavista robot ?? Allow: / User-agent: AltaVista V2.0B crawler@evreka.com Allow: / User-agent: Amiga Voyager Browser, Amiga Disallow: / User-agent: Amiga-AWeb/3.4.167SE Disallow: / User-agent: ananzi Disallow: / User-agent: AnnoMille spider 0.1 alpha - http://www.annomille.it Disallow: / User-agent: Anonymized by ProxyOS: http://www.megaproxy.com Disallow: / User-agent: Anonymizer/1.1 Disallow: / User-agent: Anthill Disallow: / User-agent: antibot-V1.1.5/i586-linux-2.2 Disallow: / User-agent: AOL Web TV Allow: / User-agent: AOL, Mac Allow: / User-agent: AOL, Win 3.x Allow: / User-agent: A-Online Search Disallow: / User-agent: Aport Disallow: / User-agent: appie 1.1 (www.walhello.com) Disallow: / User-agent: Arachnoidea (arachnoidea@euroseek.com) Disallow: / User-agent: Arachnophilia Disallow: / User-agent: Arale Disallow: / User-agent: Araneo Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: Aretha Disallow: / User-agent: ARIADNE Disallow: / User-agent: arks Disallow: / User-agent: ASpider (Associative Spider) Disallow: / User-agent: ASPSeek/1.2.4a Disallow: / User-agent: ASPSeek/1.2.5 Disallow: / User-agent: ASPSeek/1.2.6 Disallow: / User-agent: ASPSeek/1.2.8pre Disallow: / User-agent: ASPseek/1.2.9d Disallow: / User-agent: ASSORT/0.10 Disallow: / User-agent: ATN Worldwide Disallow: / User-agent: Atomz.com Search Robot Disallow: / User-agent: Atomz/1.0 Disallow: / User-agent: AURESYS Disallow: / User-agent: AV Fetch 1.0 Disallow: / User-agent: AVSearch-3.0(AltaVista/AVC) Allow: / User-agent: BackRub Disallow: / User-agent: Bannerkiller Disallow: / User-agent: Big Brother Disallow: / User-agent: BigBrother/1.6e Disallow: / User-agent: Bizbot003 Disallow: / User-agent: BizBot04 kirk.overleaf.com Disallow: / User-agent: Bjaaland Disallow: / User-agent: BlackWidow Disallow: / User-agent: Bloodhound Disallow: / User-agent: BMCLIENT Disallow: / User-agent: Borg-Bot Disallow: / User-agent: Bot mailto:craftbot@yahoo.com Disallow: / User-agent: bright.net caching robot Disallow: / User-agent: BSDSeek/1.0 Disallow: / User-agent: BSpider Disallow: / User-agent: BullsEye Disallow: / User-agent: bumblebee@relevare.com Disallow: / User-agent: BunnySlippers Disallow: / User-agent: CACTVS Chemistry Spider Disallow: / User-agent: Calif Disallow: / User-agent: CaliforniaBrownSpider Disallow: / User-agent: Cancer Information and Support International; Disallow: / User-agent: Cassandra Disallow: / User-agent: Ceramic Tile Installation Guide (http://www.floorstransformed.com) Disallow: / User-agent: cg-eye interactive Disallow: / User-agent: Checkbot Disallow: / User-agent: Checkbot/1.51 LWP/5.18 Disallow: / User-agent: Checkbot/1.6x LWP/5.46 Disallow: / User-agent: CheckUrl Disallow: / User-agent: CheckWeb Disallow: / User-agent: ChristCrawler.com Disallow: / User-agent: churl Disallow: / User-agent: cIeNcIaFiCcIoN.nEt Disallow: / User-agent: CMC/0.01 Disallow: / User-agent: ColdFusion Disallow: / User-agent: Collective Disallow: / User-agent: Combine System Disallow: / User-agent: combine/0.0 Disallow: / User-agent: Commerce Browser Center Disallow: / User-agent: ComputingSite Robi/1.0 Disallow: / User-agent: Conceptbot Disallow: / User-agent: Contact Disallow: / User-agent: contype Disallow: / User-agent: CoolBot Disallow: / User-agent: cosmos/0.8_(robot@xyleme.com) Disallow: / User-agent: cosmos/0.9_(robot@xyleme.com) Disallow: / User-agent: Crawler Disallow: / User-agent: Crawler admin@crawler.de Disallow: / User-agent: Crawler V 0.2.x admin@crawler.de Disallow: / User-agent: crawler@alexa.com Disallow: / User-agent: CrawlerBoy Pinpoint.com Disallow: / User-agent: Cuam Ver0.050bx Disallow: / User-agent: curl/7.8 (i686-pc-linux-gnu) libcurl 7.8 (OpenSSL 0.9.6) Disallow: / User-agent: curl/7.9.3 (win32) libcurl 7.9.3 Disallow: / User-agent: CurryGuide SiteScan 1.1 Disallow: / User-agent: Cusco Disallow: / User-agent: Custom Spider www.bisnisseek.com /1.0 Disallow: / User-agent: cybercity.fr user robot / faked user agent ? Disallow: / User-agent: Cyberdog/2.0 (Macintosh; 68k) Disallow: / User-agent: CyberSpyder (amant@www.cyberspyder.com) Disallow: / User-agent: CyberSpyder Link Test Disallow: / User-agent: CyberSpyder Link Test/2.1.12 (admin@mspennyworth.com) Disallow: / User-agent: DA 3.5 (www.lidan.com) Disallow: / User-agent: DA 4.0 Disallow: / User-agent: DA 4.0 (www.downloadaccelerator.com) Disallow: / User-agent: DA 5.0 Disallow: / User-agent: dallas.mt.cs.cmu.edu Disallow: / User-agent: darkwing.cadvision.com Disallow: / User-agent: DaviesBot/1.7 (www.wholeweb.net) Disallow: / User-agent: dbDig(http://www.prairielandconsulting.com) Disallow: / User-agent: DBrowse 1.4d Disallow: / User-agent: dds explorer v1.0 beta Disallow: / User-agent: DeWeb(c) Katalog/Index Disallow: / User-agent: Die Blinde Kuh Disallow: / User-agent: DienstSpider Disallow: / User-agent: diff. IPs / services (for example Phenominet.com link checking) Disallow: / User-agent: diff. IPs / services (for example: in conjunction w. Weblink's Checker) Disallow: / User-agent: different IPs using the HTTPClient library (mostly link checking) Disallow: / User-agent: Digger Disallow: / User-agent: Digger/1.0 JDK/1.3.0rc3 Disallow: / User-agent: Digimarc MarcSpider Disallow: / User-agent: Digimarc Marcspider/CGI Disallow: / User-agent: Digital Integrity Robot Disallow: / User-agent: DigOut4U Disallow: / User-agent: DIIbot/1.2 Disallow: / User-agent: Direct Hit Grabber Disallow: / User-agent: DittoSpyder Disallow: / User-agent: dlman Disallow: / User-agent: DNAbot Disallow: / User-agent: DoCoMo/1.0/N210i/c10 Disallow: / User-agent: Download Demon/3.x.x.x Disallow: / User-agent: DownLoad Express Disallow: / User-agent: Download Express 1.0 Disallow: / User-agent: download manager Disallow: / User-agent: Download Wonder Disallow: / User-agent: Downloadmanager ? Disallow: / User-agent: DragonBot Disallow: / User-agent: dsl.net user robot Disallow: / User-agent: DSurf15a VA Disallow: / User-agent: DSurf15a xx Disallow: / User-agent: dtSearchSpider Disallow: / User-agent: Dual Proxy Disallow: / User-agent: DWCP (Dridus' Web Cataloging Project) Disallow: / User-agent: EbiNess Disallow: / User-agent: EBrowse 1.4b Disallow: / User-agent: eCatch/3.0 Disallow: / User-agent: EchO!/2.0 Disallow: / User-agent: e-collector Disallow: / User-agent: EI*Net/0.1 libwww/0.1 Disallow: / User-agent: EIT Link Verifier Robot Disallow: / User-agent: ELFINBOT Disallow: / User-agent: Emacs-w3 Search Engine Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf 1.00 Disallow: / User-agent: EnigmaBot Disallow: / User-agent: Enterprise_Search/1.0 Disallow: / User-agent: EroCrawler Disallow: / User-agent: e-sense 1.0 ea(www.vigiltech.com/esensedisclaim.html) Disallow: / User-agent: ESISmartSpider Disallow: / User-agent: Esther Disallow: / User-agent: ESurf15a 15 Disallow: / User-agent: Evliya Celebi Disallow: / User-agent: Excalibur Internet Spider V6.5.4 Disallow: / User-agent: Express WebPictures (www.express-soft.com) Disallow: / User-agent: ExtractorPro Disallow: / User-agent: FairAd Client Disallow: / User-agent: fake ? Disallow: / User-agent: fake entry ?? Disallow: / User-agent: faked user agent Disallow: / User-agent: FastCrawler Disallow: / User-agent: FastCrawler 3.0.1 (crawler@1klik.dk) Disallow: / User-agent: FAST-WebCrawler/2.2.10 (Multimedia Search) (crawler@fast.no; http://www.fast. Disallow: / User-agent: FAST-WebCrawler/2.2.6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear Disallow: / User-agent: FAST-WebCrawler/2.2.7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear Disallow: / User-agent: FAST-WebCrawler/2.2.8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear Disallow: / User-agent: FAST-WebCrawler/3.2 test Disallow: / User-agent: FAST-WebCrawler/3.3 (crawler@fast.no; http://fast.no/support.php?c=faqs/crawl Disallow: / User-agent: FAST-WebCrawler/3.4/Nirvana (crawler@fast.no; http://fast.no/support.php?c=fa Disallow: / User-agent: FAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast.no/support.php? Disallow: / User-agent: FAST-WebCrawler/3.5 (atw-crawler at fast dot no; http://fast.no/support.php?c Disallow: / User-agent: FavOrg Disallow: / User-agent: Favorites Sweeper v.2.03 Disallow: / User-agent: Felix IDE Disallow: / User-agent: FetchRover Disallow: / User-agent: fido Disallow: / User-agent: FileHound 1.2 Disallow: / User-agent: Firefly/1.0 Disallow: / User-agent: Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5) Disallow: / User-agent: Fish search Disallow: / User-agent: FlashGet Disallow: / User-agent: Floortransformed.com robot (link ckecking ??) Disallow: / User-agent: Fluid Dynamics Search Engine robot Disallow: / User-agent: flunky Disallow: / User-agent: fly/6.01 libwww/4.0D Disallow: / User-agent: Fouineur Disallow: / User-agent: Freecrawl Disallow: / User-agent: FSurf15a 01 Disallow: / User-agent: FunnelWeb Disallow: / User-agent: GAIS Robot/1.0B2 Disallow: / User-agent: GaisLab data gatherer, Gaisbot/3.0+(indexer@gais.cs.ccu.edu.tw;+http://gais.c Disallow: / User-agent: gammaSpider, FocusedCrawler Disallow: / User-agent: gazz Disallow: / User-agent: gazz/2.1 (gazz@nttrd.com) Disallow: / User-agent: GCreep Disallow: / User-agent: Gecko 0.7 Linux Pentium II Disallow: / User-agent: Gecko 0.9 Linux Pentium II Disallow: / User-agent: geckobot Disallow: / User-agent: GetBot Disallow: / User-agent: GetRight/3.3.3 Disallow: / User-agent: GetRight/4.5xx Disallow: / User-agent: GetRight/4.x Disallow: / User-agent: GetterroboPlus Puu Disallow: / User-agent: GetURL Disallow: / User-agent: gigabaz/3.14 (baz@gigabaz.com; http://gigabaz.com/gigabaz/) Disallow: / User-agent: Gigabot/1.0 Disallow: / User-agent: GNODSPIDER (www.gnod.net) Disallow: / User-agent: GNU file downloader Disallow: / User-agent: Go!Zilla 3.x (www.gozilla.com) Disallow: / User-agent: Go!Zilla/4.0.0.26 Disallow: / User-agent: Go-Ahead-Got-It/1.1 Disallow: / User-agent: Golem Disallow: / User-agent: Goofer/0.2 Disallow: / User-agent: GotIt Web accelerator Disallow: / User-agent: Grapnel/0.01 Experiment Disallow: / User-agent: Griffon Disallow: / User-agent: Gromit Disallow: / User-agent: Gseek.com (site is offline) robot Disallow: / User-agent: Gulliver/1.3 Disallow: / User-agent: Gulper Bot Disallow: / User-agent: Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) Disallow: / User-agent: HamBot Disallow: / User-agent: HappyBot (gserver.kw.net) Disallow: / User-agent: Harvest Disallow: / User-agent: Harvest-NG/1.0.2 Disallow: / User-agent: havIndex Disallow: / User-agent: heraSpider Disallow: / User-agent: HI (HTML Index) Search Disallow: / User-agent: Hippias/0.9 Beta Disallow: / User-agent: HKU WWW Octopus Disallow: / User-agent: HLoader Disallow: / User-agent: Home.com user agent Disallow: / User-agent: HomePageSearch(hpsearch.uni-trier.de) Disallow: / User-agent: Hometown Spider Pro Disallow: / User-agent: HotJava browser plus HTML Component 1.1.x Disallow: / User-agent: HotJava/1.0.1/JRE1.1.x Disallow: / User-agent: ht://Dig Disallow: / User-agent: Html Link Validator (www.lithopssoft.com) Disallow: / User-agent: html_analyzer Disallow: / User-agent: HTMLgobble Disallow: / User-agent: http://@nonymouse.com/ (Linux) Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: Hyper-Decontextualizer Disallow: / User-agent: HSmShSkki Disallow: / User-agent: I, Robot Disallow: / User-agent: ia_archiver Disallow: / User-agent: ia_archiver/1.6 Disallow: / User-agent: iajaBot Disallow: / User-agent: IBM_Planetwide Disallow: / User-agent: Ibot/1.0 libwww-perl/0.40 Disallow: / User-agent: IBrowse/2.2 (AmigaOS 3.5) Disallow: / User-agent: IBrowse/2.2 (Windows 3.1) Disallow: / User-agent: iCab/2.5.2 (Macintosh; I; PPC) Disallow: / User-agent: ideare - SignSite/1.x Disallow: / User-agent: IE 2.x, Mac Power PC Disallow: / User-agent: IE 2.x, WinNT Disallow: / User-agent: IE 3.x, AOL, Win95 Disallow: / User-agent: IE 3.x, Win 3.1 Disallow: / User-agent: IE 3.x, Win95 Disallow: / User-agent: IE 3.x, WinNT Disallow: / User-agent: IE 3.x, WinXP Disallow: / User-agent: IE 4.x, AOL, Win98 Disallow: / User-agent: IE 4.x, Mac Power PC Disallow: / User-agent: IE 4.x, Win95 Disallow: / User-agent: IE 4.x, WinNT Disallow: / User-agent: IE 5 fake user agent Disallow: / User-agent: IE 5.5 Compatible Browser Disallow: / User-agent: IE 5.5, Win2000 / Jaring.my user agent Disallow: / User-agent: IE 5.5, Win2000 / user agent Disallow: / User-agent: IE 5.5, Win2000 with MS .NET SDK Disallow: / User-agent: IE 5.x, Mac PowerPC, AtHome user Disallow: / User-agent: IE PDA Browser, Windows CE Disallow: / User-agent: IEFav172Free Disallow: / User-agent: image.kapsi.net Disallow: / User-agent: Imagelock Disallow: / User-agent: IncyWincy Disallow: / User-agent: IncyWincy data gatherer(webmaster@loopimprovements.com,http://www.loopimprove Disallow: / User-agent: IncyWincy page crawler(webmaster@loopimprovements.com,http://www.loopimprovem Disallow: / User-agent: IncyWincy(http://www.look.com) Disallow: / User-agent: IncyWincy(http://www.loopimprovements.com/robot.html) Disallow: / User-agent: IncyWincy/2.1(loopimprovements.com/robot.html) Disallow: / User-agent: Inet library Disallow: / User-agent: InfoNaviRobot(F107) Disallow: / User-agent: Informant Disallow: / User-agent: InfoSeek Robot 1.0 Disallow: / User-agent: Infoseek Sidewinder Disallow: / User-agent: InfoSeek Sidewinder/0.9 Disallow: / User-agent: InfoSpiders Disallow: / User-agent: Ingrid Disallow: / User-agent: Inktomi Search Disallow: / User-agent: Inktomi Slurp Disallow: / User-agent: Inspector Web Disallow: / User-agent: InstallShield DigitalWizard Disallow: / User-agent: IntelliAgent Disallow: / User-agent: Internet Cruiser Robot Disallow: / User-agent: Internet Direct Library for Borland (used as E-Mail collector) Disallow: / User-agent: Internet Explorer 4.0 URL check Disallow: / User-agent: Internet Ninja 6.0 Disallow: / User-agent: Internet Shinchakubin Disallow: / User-agent: InternetLinkAgent/3.1 Disallow: / User-agent: InternetSeer.com Disallow: / User-agent: Iplexx Austria spider ?? Disallow: / User-agent: IRIX 6.5 Disallow: / User-agent: Iron33 Disallow: / User-agent: Israeli-search Disallow: / User-agent: ITI Spider Disallow: / User-agent: Jack Disallow: / User-agent: Janas (Ideare.com / Tiscali.it) robot Disallow: / User-agent: Java 1.1 Disallow: / User-agent: Java Development Kit Disallow: / User-agent: Java1.0.21.0 Disallow: / User-agent: Java1.1.xx.x Disallow: / User-agent: Java1.3.0rc1 Disallow: / User-agent: Java1.3.x Disallow: / User-agent: Java1.4.0 Disallow: / User-agent: JavaBee Disallow: / User-agent: JBot Java Web Robot Disallow: / User-agent: JCheckLinks Java hyperlink validator Disallow: / User-agent: JCheckLinks/0.1 RPT-HTTPClient/0.3-1 Disallow: / User-agent: JCrawler Disallow: / User-agent: JDK/1.1 Disallow: / User-agent: Jeeves Allow: / User-agent: Jeode/1.7.2 Disallow: / User-agent: JetCar Disallow: / User-agent: Jigsaw/2.2.0 W3C_CSS_Validator_JFouffa/2.0 Disallow: / User-agent: JoBo Java Web Robot Disallow: / User-agent: JoBo/@JOBO_VERSION@(http://www.matuschek.net/jobo.html) Disallow: / User-agent: Jobot Disallow: / User-agent: JoeBot Disallow: / User-agent: JumpStation Disallow: / User-agent: Katipo Disallow: / User-agent: KDD-Explorer Disallow: / User-agent: KE_1.0/2.0 libwww/5.2.8 Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Kilroy Disallow: / User-agent: KIT-Fireball Disallow: / User-agent: KIT-Fireball/2.0 Disallow: / User-agent: KIT-Fireball/2.0 (compatible; Mozilla 4.0; MSIE 5.5) Disallow: / User-agent: K-Meleon/0.6 (Windows; U; Windows NT 5.1; en-US; rv:0.9.5) Gecko/20011011 Disallow: / User-agent: KO_Yappo_Robot Disallow: / User-agent: Konqueror 2.0.x, X11 Disallow: / User-agent: Konqueror 2.1.x, X11 Disallow: / User-agent: Konqueror 2.2.x Disallow: / User-agent: Konqueror 2.2.x, Linux Disallow: / User-agent: kulturarw3/0.1 Disallow: / User-agent: LabelGrabber Disallow: / User-agent: larbin Disallow: / User-agent: larbin (samualt9@bigfoot.com) Disallow: / User-agent: larbin_2.1.1 larbin2.1.1@somewhere.com Disallow: / User-agent: larbin_2.2.0 (crawl@compete.com) Disallow: / User-agent: larbin_2.2.0 crawl@compete.com Disallow: / User-agent: larbin_2.2.1_de_Viennot (Laurent.Viennot@inria.fr) Disallow: / User-agent: larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp) Disallow: / User-agent: larbin_2.2.2 sugayama@lab7.kuis.kyoto-u.ac.jp Disallow: / User-agent: larbin_2.2.2_guillaume (guillaume@liafa.jussieu.fr) Disallow: / User-agent: larbin_2.2.2_guillaume guillaume@liafa.jussieu.fr Disallow: / User-agent: larbin_2.6.0 (larbin2.6.0@unspecified.mail) Disallow: / User-agent: larbin_2.6.1 (larbin2.6.1@unspecified.mail) Disallow: / User-agent: larbin_2.6_basileocaml (basile.starynkevitch@cea.fr) Disallow: / User-agent: larbin_devel (http://pauillac.inria.fr/~ailleret/prog/larbin/) Disallow: / User-agent: LECodeChecker/3.0 libgetdoc/1.0 Disallow: / User-agent: legs Disallow: / User-agent: LEIA/2.90 Disallow: / User-agent: LEIA/3.01pr (LEIAcrawler; [SNIP]) Disallow: / User-agent: LexiBot/1.00 Disallow: / User-agent: Libertech-Rover www.libertech.com? Disallow: / User-agent: libWeb/clsHTTP -- hiongun@kt.co.kr Disallow: / User-agent: libwww-perl/5.41 Disallow: / User-agent: libwww-perl/5.48 Disallow: / User-agent: libwww-perl/5.50 Disallow: / User-agent: libwww-perl/5.52 FP/2.1 Disallow: / User-agent: libwww-perl/5.52 FP/4.0 Disallow: / User-agent: libwww-perl/5.53 Disallow: / User-agent: libwww-perl/5.63 Disallow: / User-agent: Link checking software ? Disallow: / User-agent: Link Valet Online 1.1 Disallow: / User-agent: Link Validator Disallow: / User-agent: Linkalarm link validation Disallow: / User-agent: LinkAlarm/2.x Disallow: / User-agent: Linkbot Disallow: / User-agent: Linkguard.com link validation (service is offline) Disallow: / User-agent: linklooker Disallow: / User-agent: LinkProver 2.1 Disallow: / User-agent: Links - http://gossamer-threads.com/scripts/links/ Disallow: / User-agent: Links (0.97pre12; Linux 2.2.14-5.0 i686; 80x24) Disallow: / User-agent: Links 2.0 (http://gossamer-threads.com/scripts/links/) Disallow: / User-agent: Links SQL (http://gossamer-threads.com/scripts/links-sql/) Disallow: / User-agent: LinkScan Disallow: / User-agent: LinkScan/8.x Unix Disallow: / User-agent: LinkScan/9.0 Unix Disallow: / User-agent: LinkScan/9.0g Unix Disallow: / User-agent: LinksManager.com (http://linksmanager.com/linkchecker.html) Disallow: / User-agent: LinkWalker Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: Lockon Disallow: / User-agent: Lockstep Spider/1.0 Disallow: / User-agent: logo.gif Crawler Disallow: / User-agent: Lotus Notes 5.0 browser Disallow: / User-agent: Lotus Notes browser Disallow: / User-agent: Lotus-Notes/4.5 ( Windows-NT ) Disallow: / User-agent: LWP::Simple/5.22 Disallow: / User-agent: LWP::Simple/5.36 Disallow: / User-agent: LWP::Simple/5.50 Disallow: / User-agent: LWP::Simple/5.53 Disallow: / User-agent: LWP::Simple/5.63 Disallow: / User-agent: lwp-trivial/1.32 Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: lwp-trivial/1.35 Disallow: / User-agent: Lycos Disallow: / User-agent: Lycos_Spider_(modspider) Disallow: / User-agent: Lycos_Spider_(T-Rex) Disallow: / User-agent: Lynx 2.6 Browser Disallow: / User-agent: Lynx 2.8 Browser Disallow: / User-agent: Lynx/2.6 libwww-FM/2.14 Disallow: / User-agent: Lynx/2.8.3dev.9 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6 Disallow: / User-agent: Mac WWWWorm Disallow: / User-agent: Mag-Net Disallow: / User-agent: Magpie Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot) Disallow: / User-agent: Marvin v0.3 Disallow: / User-agent: marvin/infoseek Disallow: / User-agent: Mass Downloader 2.2 Disallow: / User-agent: Mata Hari/2.00 Disallow: / User-agent: Mattie Disallow: / User-agent: MediaCrawler-1.0 (Experimental) Disallow: / User-agent: MediaFox Disallow: / User-agent: MediaSearch/0.1 Disallow: / User-agent: MegaSheep v1.0 (www.searchuk.com internet sheep) Disallow: / User-agent: Mercator-1.x Disallow: / User-agent: Mercator-2.0 Disallow: / User-agent: Mercator-Scrub-1.1 Disallow: / User-agent: Merritt/1.0 Disallow: / User-agent: MerzScope Disallow: / User-agent: MetaGer-LinkChecker Disallow: / User-agent: MetaProducts Download Express/1.x Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: Microsoft URL Control - 6.00.8xxx Disallow: / User-agent: Microsoft.com user agent Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: MindCrawler Disallow: / User-agent: Mindspring.com user robot Disallow: / User-agent: Mister PiX version.dll Disallow: / User-agent: mnoGoSearch search engine software Disallow: / User-agent: Mobile phone browser Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 (moget@goo.ne.jp) Disallow: / User-agent: MOMspider Disallow: / User-agent: Monster Disallow: / User-agent: Motor Disallow: / User-agent: Mozi! Disallow: / User-agent: Mozilla Disallow: / User-agent: Mozilla (Mozilla@somewhere.com) Disallow: / User-agent: Mozilla/1.1 (compatible; MSPIE 2.0; Windows CE) Disallow: / User-agent: Mozilla/1.22 (compatible; MSIE 2.0d; Windows NT) Disallow: / User-agent: Mozilla/2.0 (compatible; AOL 3.0; Mac_PowerPC) Disallow: / User-agent: Mozilla/2.0 (Compatible; AOL-IWENG 3.0; Win16) Disallow: / User-agent: Mozilla/2.0 (compatible; Ask Jeeves) Allow: / User-agent: Mozilla/2.0 (compatible; EZResult -- Internet Search Engine) Disallow: / User-agent: Mozilla/2.0 (compatible; MS FrontPage 3.0) Disallow: / User-agent: Mozilla/2.0 (compatible; MS FrontPage 4.0) Disallow: / User-agent: Mozilla/2.0 (compatible; MSIE 2.1; Mac_PowerPC) Disallow: / User-agent: Mozilla/2.0 (compatible; MSIE 3.0; AK; Windows 95) Disallow: / User-agent: Mozilla/2.0 (compatible; MSIE 3.0; Windows 3.1) Disallow: / User-agent: Mozilla/2.0 (compatible; MSIE 3.02; Update a; AK; Windows NT) Disallow: / User-agent: Mozilla/2.0 (compatible; MSIE 3.02; Update a; AOL 3.0; Windows 95) Disallow: / User-agent: Mozilla/2.0 (compatible; MSIE 3.0B; Win32) Disallow: / User-agent: Mozilla/2.0 (compatible; NEWT ActiveX; Win32) Disallow: / User-agent: Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) Disallow: / User-agent: Mozilla/2.0 compatible; Check&Get 1.1x (Windows 98) Disallow: / User-agent: Mozilla/2.01 (Win16; I) Disallow: / User-agent: Mozilla/2.02Gold (Win95; I) Disallow: / User-agent: Mozilla/3.0 (compatible) Disallow: / User-agent: Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info Disallow: / User-agent: Mozilla/3.0 (compatible; Indy Library) Disallow: / User-agent: Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com) Disallow: / User-agent: Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk) Disallow: / User-agent: Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com) Disallow: / User-agent: Mozilla/3.0 (compatible; NetPositive/2.2) Disallow: / User-agent: Mozilla/3.0 (compatible; Opera/3.0; Windows 3.1) v3.1 Disallow: / User-agent: Mozilla/3.0 (compatible; Opera/3.0; Windows 95/NT4) 3.2 Disallow: / User-agent: Mozilla/3.0 (compatible; REL Software, Web Link Validator 2.x) Disallow: / User-agent: Mozilla/3.0 (compatible; Web Link Validator 2.x) Disallow: / User-agent: Mozilla/3.0 (compatible; WebCapture 2.0; Auto; Windows) Disallow: / User-agent: Mozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56) Disallow: / User-agent: Mozilla/3.0 (Compatible;Viking/1.8) Disallow: / User-agent: Mozilla/3.0 (Liberate DTV 1.1) Disallow: / User-agent: Mozilla/3.0 (Planetweb/2.100 JS SSL US; Dreamcast US) Disallow: / User-agent: Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.ht Disallow: / User-agent: Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Disallow: / User-agent: Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Disallow: / User-agent: Mozilla/3.0 (Vagabondo/1.x MT; webagent@wise-guys.nl; http://webagent.wise-gu Disallow: / User-agent: Mozilla/3.0 (Win16; I) Disallow: / User-agent: Mozilla/3.0 (Win95; I) Disallow: / User-agent: Mozilla/3.0 (WinNT; I) Disallow: / User-agent: Mozilla/3.0 (WorldGate Gazelle 3.5.1 build 11; FreeBSD2.2.8-STABLE) Disallow: / User-agent: Mozilla/3.0 (X11; I; OSF1 V4.0 alpha) Disallow: / User-agent: Mozilla/3.0 NAVIO_AOLTV (11; 13; Philips; PH200; 1; R2.0C36_AOL.0110OPTIK; R2 Disallow: / User-agent: Mozilla/3.0 WebTV/1.2 (compatible; MSIE 2.0) Disallow: / User-agent: Mozilla/3.01 (compatible; AmigaVoyager/2.95; AmigaOS/MC680x0) Disallow: / User-agent: Mozilla/3.01 (Compatible; Links2Go Similarity Engine) Disallow: / User-agent: Mozilla/3.01 (compatible; Netbox/3.5 R92; Linux 2.2) Disallow: / User-agent: Mozilla/3.01-C-MACOS8 (Macintosh; I; PPC) Disallow: / User-agent: Mozilla/3.01Gold (X11; I; Linux 2.0.32 i486) Disallow: / User-agent: Mozilla/3.01Gold (X11; I; SunOS 5.5.1 sun4m) Disallow: / User-agent: Mozilla/3.01SGoldC-SGI (X11; I; IRIX 6.3 IP32) Disallow: / User-agent: Mozilla/3.04 (compatible; ANTFresco/2.13; RISC OS 4.02) Disallow: / User-agent: Mozilla/3.04 (compatible; NCBrowser/2.35; ANTFresco/2.17; RISC OS-NC 5.13 Laz Disallow: / User-agent: Mozilla/3.x (I-Opener 1.1; Netpliance) Disallow: / User-agent: Mozilla/4.0 (compatible; Advanced Email Extractor v2.xx) Disallow: / User-agent: Mozilla/4.0 (compatible; BorderManager 3.0) Disallow: / User-agent: Mozilla/4.0 (compatible; DnloadMage 1.0) Disallow: / User-agent: Mozilla/4.0 (compatible; FastCrawler3, support-fastcrawler3@fast.no) Disallow: / User-agent: Mozilla/4.0 (compatible; FDSE robot) Disallow: / User-agent: Mozilla/4.0 (compatible; grub-client-0.2.x; Crawl your stuff with http://grub Disallow: / User-agent: Mozilla/4.0 (compatible; grub-client-0.3.x; Crawl your own stuff with http:// Disallow: / User-agent: Mozilla/4.0 (compatible; ICS 1.2.xxx) Disallow: / User-agent: Mozilla/4.0 (compatible; Iplexx Spider/1.0, http://www.iplexx.at) Disallow: / User-agent: Mozilla/4.0 (compatible; Lotus-Notes/5.0; Windows-NT) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indones Allow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) WebQues Allow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Snap.home.1) Allow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.01; AOL 4.0; Windows 98) Allow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Mac_PowerPC) Allow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.01; MSIECrawler; Windows 95) Allow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows 95) Allow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows 95; Snap.home.5) Allow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; MSN Companion 2.0; 800x600; C Allow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT Windows CE) Allow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT) Allow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot) Microsof Allow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; AOL 5.0; Windows 95; DigExt; Gateway2000; Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC; AtHome021) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Win32) via proxy gateway CERN-HTTPD/3.0 li Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; KITV4.7 Wanadoo) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en] Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; www.psychedelix.com) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows 95) via Avirt Gateway Server Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; Madasafish) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) (samualt9@bigfoot.com) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; NetCaptor 6.5.0RC1) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; AOL 4.0; Windows 98; GoBeez (www.gobeez.co Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 95) via proxy gateway CERN-HTTPD/3 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 95; Transmission Segment; Hotbar 2 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; KITV4.7 Wanadoo) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SAFEXPLORER TL) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SYMPA; Katiesoft 7; SimulBrows Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; BTinternet V8.1) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MSIECrawler) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; QXW03018) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Active Cache Request Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; .NET CLR 1.0.3705) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; AspTear 1.5) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; N_o_k_i_a) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32) WebWasher 3.0 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) REL Software, Web Link Validat Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) Web Link Validator 2.x) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Net M@nager V3.02 - www.vinn.c Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 3.0) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; IOpener Release 1 Disallow: / User-agent: Mozilla/4.0 (compatible; Netcraft Web Server Survey) Disallow: / User-agent: Mozilla/4.0 (compatible; Opera/3.0; Windows 4.10) 3.51 [en] Disallow: / User-agent: Mozilla/4.0 (compatible; Powermarks/3.5; Windows 95/98/2000/NT) Disallow: / User-agent: Mozilla/4.0 (compatible; SiteKiosk 4.0; MSIE 5.0; Windows 98; SiteCoach 1.0) Disallow: / User-agent: Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5) Disallow: / User-agent: Mozilla/4.0 (compatible; www.galaxy.com) Disallow: / User-agent: Mozilla/4.0 (compatible; www.linkguard.com Linkguard Online 1.0; Windows NT) Disallow: / User-agent: Mozilla/4.0 (fantomBrowser) Disallow: / User-agent: Mozilla/4.0 (fantomCrew Browser) Disallow: / User-agent: Mozilla/4.0 (hhjhj@yahoo.com) Disallow: / User-agent: Mozilla/4.0 (Sleek Spider/1.2) Disallow: / User-agent: Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutb Disallow: / User-agent: Mozilla/4.0 compatible ZyBorg/1.0 for Homepage (ZyBorg@WISEnutbot.com; http:/ Disallow: / User-agent: Mozilla/4.0 WebTV/2.6 (compatible; MSIE 4.0) Disallow: / User-agent: Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_TrueRobot/1.4 libwww/5.2.8 Disallow: / User-agent: Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_VoilaBot/1.6 libwww/5.3.2 Disallow: / User-agent: Mozilla/4.01 [en](Win95;I) Disallow: / User-agent: Mozilla/4.02 [en] (X11; I; SunOS 5.6 sun4u) Disallow: / User-agent: Mozilla/4.04 [en] (X11; I; HP-UX B.10.20 9000/712) Disallow: / User-agent: Mozilla/4.04 [en] (X11; I; IRIX 5.3 IP22) Disallow: / User-agent: Mozilla/4.05 (Macintosh; I; 68K, Nav) Disallow: / User-agent: Mozilla/4.05 (Macintosh; I; PPC, Nav) Disallow: / User-agent: Mozilla/4.05 [en] (X11; I; SunOS 4.1.4 sun4m) Disallow: / User-agent: Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98) Disallow: / User-agent: Mozilla/4.5 (compatible; iCab 2.5.3; Macintosh; I; PPC) Disallow: / User-agent: Mozilla/4.5 (compatible; OmniWeb/4.0.5; Mac_PowerPC) Disallow: / User-agent: Mozilla/4.5 (compatible; OmniWeb/4.1-beta-1; Mac_PowerPC) Disallow: / User-agent: Mozilla/4.5 [en]C-CCK-MCD {RuralNet} (Win98; I) Disallow: / User-agent: Mozilla/4.5 RPT-HTTPClient/0.3-2 Disallow: / User-agent: Mozilla/4.5b1 [en] (X11; I; Linux 2.0.35 i586) Disallow: / User-agent: Mozilla/4.61 [de] (OS/2; I) Disallow: / User-agent: Mozilla/4.7 (compatible; http://eidetica.com/spider) Disallow: / User-agent: Mozilla/4.7 (compatible; OffByOne; Windows 98) Webster Pro V3.2 Disallow: / User-agent: Mozilla/4.7 (compatible; Whizbang) Disallow: / User-agent: Mozilla/4.7 (compatible; WhizBang; http://www.whizbang.com/crawler) Disallow: / User-agent: Mozilla/4.7 [en]C-CCK-MCD {BresnanLink} (Win95; U) Disallow: / User-agent: Mozilla/4.7 [en]C-CCK-MCD {Yahoo;YIP052400} (Win95; I) Allow: / User-agent: Mozilla/4.72 [en] (BACS http://www.ba.be) Disallow: / User-agent: Mozilla/4.72C-CCK-MCD Caldera Systems OpenLinux [en] (X11; U; Linux 2.2.14 i6 Disallow: / User-agent: Mozilla/4.75C-ja [ja] (X11; U; OSF1 V5.1 alpha) Disallow: / User-agent: Mozilla/4.76 (Windows 98; U) Opera 5.12 [en] Disallow: / User-agent: Mozilla/4.76 [en] (X11; U; FreeBSD 4.4-STABLE i386) Disallow: / User-agent: Mozilla/4.76 [en] (X11; U; SunOS 5.7 sun4u) Disallow: / User-agent: Mozilla/4.77C-SGI [en] (X11; U; IRIX 6.5 IP32) Disallow: / User-agent: Mozilla/5.0 Disallow: / User-agent: Mozilla/5.0 (compatible; Konqueror/2.0.1; X11); Supports MD5-Digest; Supports Disallow: / User-agent: Mozilla/5.0 (compatible; Konqueror/2.1.1; X11) Disallow: / User-agent: Mozilla/5.0 (compatible; Konqueror/2.2.2) Disallow: / User-agent: Mozilla/5.0 (compatible; Konqueror/2.2.2; Linux 2.4.14-xfs; X11; i686) Disallow: / User-agent: Mozilla/5.0 (Macintosh; U; PPC; en-US; rv:0.9.2) Gecko/20010726 Netscape6/6.1 Disallow: / User-agent: Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Disallow: / User-agent: Mozilla/5.0 (SunOS 5.8 sun4u; U) Opera 5.0 [en] Disallow: / User-agent: Mozilla/5.0 (Windows; U; Win98; en-US; rv:0.9.2) Gecko/20010726 Netscape6/6.1 Disallow: / User-agent: Mozilla/5.0 (X11; U; Linux 2.4.2-2 i586; en-US; m18) Gecko/20010131 Netscape6 Disallow: / User-agent: Mozilla/5.0 (X11; U; Linux 2.4.2-2 i686; en-US; 0.7) Gecko/20010316 Disallow: / User-agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.3) Gecko/20010801 Disallow: / User-agent: Mozilla/5.0 Galeon/1.0.2 (X11; Linux i686; U;) Gecko/20011224 Disallow: / User-agent: Mozilla/5.0 URL-Spider Disallow: / User-agent: Mozilla/5.0 usww.com-Spider-for-w8.net Disallow: / User-agent: Mozilla/6.01 (compatible; noydb v0.42) Disallow: / User-agent: MS Frontpage 4.x Disallow: / User-agent: MSFrontPage/4.0 Disallow: / User-agent: MSIE 4.0 (Win95) Disallow: / User-agent: MSProxy/2.0 Disallow: / User-agent: multiBlocker browser Disallow: / User-agent: MultiText/0.1 Disallow: / User-agent: Muncher Disallow: / User-agent: murph.cais.net (rapid fire... sigh) Disallow: / User-agent: Muscat Ferret Disallow: / User-agent: Mwd.Search Disallow: / User-agent: MyCNNSpider Disallow: / User-agent: myDaemon Disallow: / User-agent: MyGetRight/1.0.0 Disallow: / User-agent: MyGetRight/1.0b Disallow: / User-agent: NABOT/5.0 Disallow: / User-agent: nabot_1.0 Disallow: / User-agent: NationalDirectoryAddURL/1.0 Disallow: / User-agent: NationalDirectory-WebSpider/1.3 Disallow: / User-agent: NCBrowser, Risc OS Disallow: / User-agent: NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html) Disallow: / User-agent: NEC Research Agent -- compuman at research.nj.nec.com Disallow: / User-agent: NEC-Hayek/1.0 Disallow: / User-agent: NEC-MeshExplorer Disallow: / User-agent: Nederland.zoek Disallow: / User-agent: NetAnts/1.2x Disallow: / User-agent: NetCarta WebMap Engine Disallow: / User-agent: NETCOMplete/x.xx Disallow: / User-agent: NetLookout/2.24 Disallow: / User-agent: NetMechanic Disallow: / User-agent: NetMechanic Vx.0 Disallow: / User-agent: NetPositive BEOS browser Disallow: / User-agent: Netprospector JavaCrawler Disallow: / User-agent: NetResearchServer/2.2(loopimprovements.com/robot.html) Disallow: / User-agent: Netscape 2.x Gold, Win95 Disallow: / User-agent: Netscape 2.x, Win3.x, International Disallow: / User-agent: Netscape 3.x, FreeBSD Disallow: / User-agent: Netscape 3.x, Irix Disallow: / User-agent: Netscape 3.x, Linux Disallow: / User-agent: Netscape 3.x, Mac Disallow: / User-agent: Netscape 3.x, OSF1 V4.0 alpha Disallow: / User-agent: Netscape 3.x, Risc OS Disallow: / User-agent: Netscape 3.x, SunOS Disallow: / User-agent: Netscape 3.x, Win3.x Disallow: / User-agent: Netscape 3.x, Win95 Disallow: / User-agent: Netscape 3.x, WinNT Disallow: / User-agent: Netscape 4.7x Japan OSF1 alpha Disallow: / User-agent: Netscape 4.7x, FreeBSD Disallow: / User-agent: Netscape 4.7x, SunOS Disallow: / User-agent: Netscape 4.x HP-Unix Disallow: / User-agent: Netscape 4.x IRIX Disallow: / User-agent: Netscape 4.x Macintosh 68k Disallow: / User-agent: Netscape 4.x Macintosh PowerPC Disallow: / User-agent: Netscape 4.x SunOS 4.1.4 Disallow: / User-agent: Netscape 4.x SunOS 5.6 Disallow: / User-agent: Netscape 4.x, Linux Disallow: / User-agent: Netscape 4.x, OS/2 Disallow: / User-agent: Netscape 6.x, Linux, Pentium I Disallow: / User-agent: Netscape 6.x, Mac PowerPC Disallow: / User-agent: Netscape 6.x, Win98 Disallow: / User-agent: NetScoop Disallow: / User-agent: Netscpape 4.7x, Caldera Open Linux, Pentium III Disallow: / User-agent: NetZippy Disallow: / User-agent: NeuralBot/0.2 Disallow: / User-agent: newscan-online Disallow: / User-agent: NHSE Web Forager Disallow: / User-agent: NICO/1.0 Disallow: / User-agent: Noago Spider Disallow: / User-agent: Nocilla/1.0 Disallow: / User-agent: Nokia.com network Disallow: / User-agent: Nokia7110/1.0 (05.01) (Google WAP Proxy/1.0) Allow: / User-agent: Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com Allow: / User-agent: Nomad Disallow: / User-agent: Northern Light Gulliver Disallow: / User-agent: Novell Border Manager security suite Disallow: / User-agent: nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp) Disallow: / User-agent: nttrd.com / Infobee.ne.jp robot Disallow: / User-agent: nzexplorer Disallow: / User-agent: oBot ((compatible;Win32)) Disallow: / User-agent: Occam Disallow: / User-agent: Octopus Disallow: / User-agent: Offline Explorer 1.* Disallow: / User-agent: OGspider Disallow: / User-agent: OliverPerry Disallow: / User-agent: OmniWeb 4.0.x Mac browser Disallow: / User-agent: OmniWeb 4.1.x Mac browser Disallow: / User-agent: Open Text Index Robot Disallow: / User-agent: Openfind data gatherer Disallow: / User-agent: Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://w Disallow: / User-agent: Openfind Robot/1.1A2 Disallow: / User-agent: OpenTextSiteCrawler/2.9.2 Disallow: / User-agent: Opera 3.x WinNT Disallow: / User-agent: Opera 3.x, Win3.x Disallow: / User-agent: Opera 3.x, Win95/NT Disallow: / User-agent: Opera 5.0 Linux Disallow: / User-agent: Opera 5.11 faked WinME referer Disallow: / User-agent: Opera 5.12 Win98 Disallow: / User-agent: Opera 5.x, SunOS Disallow: / User-agent: Opera 5.x, Win 98 Disallow: / User-agent: Opera 6.0 WinNT Disallow: / User-agent: Opera 6.x Linux Disallow: / User-agent: Opera/5.0 (Linux 2.0.38 i386; U) [en] Disallow: / User-agent: Opera/5.11 (Windows ME; U) [ru] Disallow: / User-agent: Opera/5.12 (Windows 98; U) [en] Disallow: / User-agent: Opera/6.0 (Linux 2.4.8-26mdk i686; U) [en] Disallow: / User-agent: Opera/6.0 (Windows NT 4.0; U) [de] Disallow: / User-agent: Oracle iMTCrawler Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: Orb Search Disallow: / User-agent: Pack Rat Disallow: / User-agent: Packard Bell Net user robot (*not* Google) Disallow: / User-agent: PageBoy Disallow: / User-agent: PagmIEDownload Disallow: / User-agent: parallelContextFocusCrawler1.1parallelContextFocusCrawler1.1 Disallow: / User-agent: ParaSite Disallow: / User-agent: ParaSite/1.0b (http://www.ianett.com/parasite/) Disallow: / User-agent: Patric Disallow: / User-agent: pavuk/0.9pl29b i686-pc-linux-gnu Disallow: / User-agent: PBrowse 1.4b Disallow: / User-agent: PDA Pocket IE 2.x, Windows CE Disallow: / User-agent: pegasus Disallow: / User-agent: PerlCrawler 1.0 Disallow: / User-agent: PEval 1.4b Disallow: / User-agent: PGP Key Agent Disallow: / User-agent: Phantom Disallow: / User-agent: PHP/3.0.16 Disallow: / User-agent: PHP/4.0.4pl1 Disallow: / User-agent: PHP/4.0.6 Disallow: / User-agent: PHP/4.1.1 Disallow: / User-agent: PHP/4.1.2 Disallow: / User-agent: PhpDig Disallow: / User-agent: PicoSearch/1.0 Disallow: / User-agent: PiltdownMan Disallow: / User-agent: Pimptrain.com's robot Disallow: / User-agent: PingALink Monitoring Services 1.0 Disallow: / User-agent: PingALink Monitoring Services 1.0 (http://www.pingalink.com) Disallow: / User-agent: Pioneer Disallow: / User-agent: Pizilla++ ver 2.45 Disallow: / User-agent: PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com) Disallow: / User-agent: PlantyNet_WebRobot_V1.9 dhkang@plantynet.com Disallow: / User-agent: PlumtreeWebAccessor Disallow: / User-agent: Pompos/1.0 pompos@iliad.fr Disallow: / User-agent: Poppi Disallow: / User-agent: Popular Iconoclast Disallow: / User-agent: Portal Juice Spider Disallow: / User-agent: PortalB Spider Disallow: / User-agent: processor/0.0ALPHA libwww-perl/0.20 Disallow: / User-agent: Progressive Download Disallow: / User-agent: Progressive Download HTTP check Disallow: / User-agent: Proxy server Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 (+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSurf15a 11 Disallow: / User-agent: PSurf15a 51 Disallow: / User-agent: PSurf15a VA Disallow: / User-agent: PureSight Disallow: / User-agent: Python-urllib/2.0a1 Disallow: / User-agent: Quarterdecks WebCompass search tool Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Rainbot1.1 Disallow: / User-agent: ramBot xtreme x.x Disallow: / User-agent: Raven Search Disallow: / User-agent: RBSE Spider Disallow: / User-agent: rcn.com user agent ? Disallow: / User-agent: RealDownload/4.0.0.4x Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: Resume Robot Disallow: / User-agent: RMA/1.0 (compatible; RealMedia) Disallow: / User-agent: Road Runner: The ImageScape Robot Disallow: / User-agent: RoadHouse Crawling System Disallow: / User-agent: Robbie the Robot Disallow: / User-agent: RoboCrawl Spider Disallow: / User-agent: RoboFox Disallow: / User-agent: Robot Francoroute Disallow: / User-agent: Robot@SuperSnooper.Com Disallow: / User-agent: Robozilla Disallow: / User-agent: Robozilla/1.0 Disallow: / User-agent: Rotondo/3.1 libwww/5.3.1 Disallow: / User-agent: Roverbot Disallow: / User-agent: RPT-HTTPClient/0.3-x Disallow: / User-agent: RRC (crawler_admin@bigfoot.com) Disallow: / User-agent: RRC crawler_admin@bigfoot.com Disallow: / User-agent: RSurf15a 41 Disallow: / User-agent: RSurf15a 51 Disallow: / User-agent: RSurf15a 81 Disallow: / User-agent: RuLeS Disallow: / User-agent: SafetyNet Robot Disallow: / User-agent: SandBot Disallow: / User-agent: Science Traveller International 1X/1.0 Disallow: / User-agent: Scooter Disallow: / User-agent: Scooter/1.0 scooter@pa.dec.com Disallow: / User-agent: Scooter/1.1 (custom) Disallow: / User-agent: Scooter/2.0 G.R.A.B. V1.1.0 Disallow: / User-agent: Scooter/2.0 G.R.A.B. X2.0 Disallow: / User-agent: Scooter_bh0-3.0.3 Disallow: / User-agent: Scooter_trk3-3.0.3 Disallow: / User-agent: Scooter2_Mercator_x-x.0 Disallow: / User-agent: Scooter-3.0.EU Disallow: / User-agent: Scooter-3.0.FS Disallow: / User-agent: Scooter-3.0.HD Disallow: / User-agent: Scooter-3.0.VNS Disallow: / User-agent: Scooter-3.0QI Disallow: / User-agent: Scooter-3.2 Disallow: / User-agent: Scooter-3.2.DIL Disallow: / User-agent: Scooter-3.2.EX Disallow: / User-agent: Scooter-3.2.JT Disallow: / User-agent: Scooter-3.2.NIV Disallow: / User-agent: Scooter-3.2.snippet Disallow: / User-agent: Scooter-3.3dev Disallow: / User-agent: Scooter-ARS-1.1-ih Disallow: / User-agent: scooter-venus-3.0.vns Disallow: / User-agent: Scooter-W3.1.2 Disallow: / User-agent: Scooter-W3-1.0 Disallow: / User-agent: ScoutAbout Disallow: / User-agent: Search.Aus-AU.COM Disallow: / User-agent: search.ch V1.4 Disallow: / User-agent: search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch) Disallow: / User-agent: Search+ Disallow: / User-agent: SearchExpress Spider0.99 Disallow: / User-agent: SearchProcess Disallow: / User-agent: SearchTone2.0 - IDEARE Disallow: / User-agent: secure-netz.de link checking (in conjunction w. LWP::Simple/5.50) Disallow: / User-agent: Senrigan Disallow: / User-agent: SG-Scout Disallow: / User-agent: ShagSeeker Disallow: / User-agent: Shai'Hulud Disallow: / User-agent: Sift Disallow: / User-agent: Simmany Robot Ver1.0 Disallow: / User-agent: Site Searcher Disallow: / User-agent: Site Valet Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer. Disallow: / User-agent: SiteSnagger Disallow: / User-agent: SiteTech-Rover Disallow: / User-agent: Skymob.com Disallow: / User-agent: Slarp/0.1 Disallow: / User-agent: SLCrawler Disallow: / User-agent: Sleek Disallow: / User-agent: Slider_Search_v1-de Disallow: / User-agent: SlowBot Disallow: / User-agent: Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html) Disallow: / User-agent: Slurp/2.0-KiteWeekly (slurp@inktomi.com; http://www.inktomi.com/slurp.html) Disallow: / User-agent: Slurp/si (slurp@inktomi.com; http://www.inktomi.com/slurp.html) Disallow: / User-agent: SlySearch (slysearch@slysearch.com) Disallow: / User-agent: Smart Spider Disallow: / User-agent: SmartDownload/1.2.67 (Win32; Jan 12 1999) Disallow: / User-agent: SmartDownload/1.2.77 (Win32; Feb 1 2000) Disallow: / User-agent: SmartDownload/1.2.77 (Win32; Jun 19 2001) Disallow: / User-agent: Snooper Disallow: / User-agent: Solbot Disallow: / User-agent: some download agent, in conjunction w. "dlman" Disallow: / User-agent: some download agent, in conjunction w. "Wildsoft Surfer" Disallow: / User-agent: Some download manager Disallow: / User-agent: Some download manager (spoofed user-agent) Disallow: / User-agent: Some download manager spoofing as MS Frontpage 3.x Disallow: / User-agent: Some download manager spoofing Netscape 4.01 Disallow: / User-agent: Some private robot (Wanadoo.fr client) Disallow: / User-agent: Spanner Disallow: / User-agent: speedfind ramBot xtreme 8.1 Disallow: / User-agent: Speedy Spider Disallow: / User-agent: Spida/0.1 Disallow: / User-agent: spider.batsch.com Disallow: / User-agent: spider.yellopet.com - www.yellopet.com Disallow: / User-agent: Spider/maxbot.com admin@maxbot.com Disallow: / User-agent: spider_monkey Disallow: / User-agent: SpiderBot Disallow: / User-agent: Spiderline Crawler Disallow: / User-agent: SpiderMan Disallow: / User-agent: SpiderView(tm) Disallow: / User-agent: Spinne/2.0 med Disallow: / User-agent: Spry Wizard Robot Disallow: / User-agent: spyder3.microsys.com Disallow: / User-agent: SQ Webscanner Disallow: / User-agent: Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu) Disallow: / User-agent: Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu) Disallow: / User-agent: Sqworm/2.9.89-BETA (beta_release; 20020130-839; i686-pc-linux-gnu) Disallow: / User-agent: SSurf15a 11 Disallow: / User-agent: Stamina/1.4 Disallow: / User-agent: Star Downloader Disallow: / User-agent: StatFetcher/1.0 Disallow: / User-agent: Suke Disallow: / User-agent: SummyCrawler Disallow: / User-agent: suntek search engine Disallow: / User-agent: SuperBot/2.6 (Win32) Disallow: / User-agent: Surfbot Disallow: / User-agent: SurferF3 1/0 Disallow: / User-agent: suzuran Disallow: / User-agent: Sven Disallow: / User-agent: swbot/0.9c libwww/5.3.1 Disallow: / User-agent: SyncBot Disallow: / User-agent: SyncIT link validation Disallow: / User-agent: SyncIT/1.x Disallow: / User-agent: SyncIT/2.x Disallow: / User-agent: Szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl Disallow: / User-agent: TACH Black Widow Disallow: / User-agent: Tarantula Disallow: / User-agent: targetblaster.com/0.9k Disallow: / User-agent: tarspider Disallow: / User-agent: Tcl W3 Robot Disallow: / User-agent: TE Disallow: / User-agent: TeacherSoft/1.0 libwww/2.17 Disallow: / User-agent: TeamSoft WinInet Component Disallow: / User-agent: TechBOT Disallow: / User-agent: telefonica.es user robot Disallow: / User-agent: Teleport Pro/1.29.1xxx Disallow: / User-agent: Templeton Disallow: / User-agent: teomaagent crawler-admin@teoma.com Disallow: / User-agent: teomaagent1 [crawler-admin@teoma.com] Disallow: / User-agent: TeomaTechnologies Disallow: / User-agent: The Expert HTML Source Viewer (http://www.expert-html.com) Disallow: / User-agent: The Jubii Indexing Robot Disallow: / User-agent: The NorthStar Robot Disallow: / User-agent: The NWI Robot Disallow: / User-agent: The Peregrinator Disallow: / User-agent: The Python Robot Disallow: / User-agent: The TkWWW Robot Disallow: / User-agent: The Web Moose Disallow: / User-agent: The Web Wombat Disallow: / User-agent: The Webfoot Robot Disallow: / User-agent: the World Wide Web Wanderer Disallow: / User-agent: The World Wide Web Worm Disallow: / User-agent: Time Warner Telecom user robot ? Disallow: / User-agent: TITAN Disallow: / User-agent: TitIn Disallow: / User-agent: tivraSpider/1.0 (crawler@tivra.com) Disallow: / User-agent: TJG/Spider Disallow: / User-agent: TLSpider Disallow: / User-agent: Toutatis 2.5-2 Disallow: / User-agent: Trampelpfad-Spider Disallow: / User-agent: Trampelpfad-Spider-v0.1 Disallow: / User-agent: tricosMetaCheck 1.2216-08-1999 (http://www.tricos.com/metacheck) Disallow: / User-agent: TSurf15a 11 Disallow: / User-agent: Twotrees Reactive Filter V2.0 Disallow: / User-agent: UCmore Disallow: / User-agent: UCMore Crawler App Disallow: / User-agent: UCSD Crawl Disallow: / User-agent: UDM Disallow: / User-agent: UdmSearch Disallow: / User-agent: UdmSearch/3.1.x Disallow: / User-agent: UIowaCrawler/1.0 Disallow: / User-agent: Ultimate Search / Smartdesk (no website) robot Disallow: / User-agent: Ultraseek Disallow: / User-agent: unet.ca Disallow: / User-agent: unknown link-checking Disallow: / User-agent: Unknown robots from 64.5.216.xx, 66.98.36.xx, 148.74.255.xx Disallow: / User-agent: unknown user robot (24.124.34.42) Disallow: / User-agent: unknown.Level3.net robot Disallow: / User-agent: unnamed Disallow: / User-agent: UP.Browser/3.01-IG01 UP.Link/3.2.3.4 Disallow: / User-agent: URL Check Disallow: / User-agent: URL Spider Pro Disallow: / User-agent: URLGetFile Disallow: / User-agent: user agent Disallow: / User-agent: user agent (HTTP header f. transferencoding) Disallow: / User-agent: user agent, looks for form-mail components (spam-bot) Disallow: / User-agent: user faked referer Disallow: / User-agent: UtilMind HTTPGet Disallow: / User-agent: Utopia WebWasher 3.0 Disallow: / User-agent: uu.net user robot/agent Disallow: / User-agent: Vagabondo/1.x MT (webagent@wise-guys.nl) Disallow: / User-agent: Valkyrie Disallow: / User-agent: Verticrawl Disallow: / User-agent: Victoria Disallow: / User-agent: vision-search Disallow: / User-agent: Voyager Disallow: / User-agent: vspider/3.x Disallow: / User-agent: VWbot Disallow: / User-agent: w@pSpider by wap4.com Disallow: / User-agent: W3C_Validator/1.122 libwww-perl/5.50 Disallow: / User-agent: w3index Disallow: / User-agent: w3m/0.1.xx Disallow: / User-agent: W3M2 Disallow: / User-agent: w3mir Disallow: / User-agent: waldec.com Disallow: / User-agent: Walhello appie Disallow: / User-agent: WallPaper Disallow: / User-agent: Wanadoo Internet provider Disallow: / User-agent: Wanadoo Internet services Disallow: / User-agent: WDG_Validator/1.1 Disallow: / User-agent: Web Core / Roots Disallow: / User-agent: Web Image Collector Disallow: / User-agent: Web Link Validator 1.5 Disallow: / User-agent: WebBandit Web Spider Disallow: / User-agent: webbandit/4.45.0 Disallow: / User-agent: WebCatcher Disallow: / User-agent: webcollage/1.7x Disallow: / User-agent: webcollage/1.8x Disallow: / User-agent: WebCollector Disallow: / User-agent: WebCompass 2.0 Disallow: / User-agent: WebCopier v2.5 Disallow: / User-agent: WebCopier v2.7a Disallow: / User-agent: WebCopy Disallow: / User-agent: webfetcher Disallow: / User-agent: WebFindBot(http://www.web-find.com) Disallow: / User-agent: webhack Disallow: / User-agent: weblayers Disallow: / User-agent: WebLinker Disallow: / User-agent: Weblink's checker/ Disallow: / User-agent: Weblog Monitor Disallow: / User-agent: WebMirror Disallow: / User-agent: WebQuest Disallow: / User-agent: WebReaper Disallow: / User-agent: WebReaper vx.x - www.webreaper.net Disallow: / User-agent: webs Disallow: / User-agent: Websnarf Disallow: / User-agent: WebSpider Disallow: / User-agent: Webspinne/1.0 webmaster@webspinne.de Disallow: / User-agent: WebStolperer Disallow: / User-agent: WebStripper/2.0x Disallow: / User-agent: WebTrends/3.0 (WinNT) Disallow: / User-agent: WebTV Disallow: / User-agent: WebVac Disallow: / User-agent: webwalk Disallow: / User-agent: WebWalker Disallow: / User-agent: WebWatch Disallow: / User-agent: WebZinger Disallow: / User-agent: WebZIP/4.1 (http://www.spidersoft.com) Disallow: / User-agent: WFARC Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.x Disallow: / User-agent: whatUseek Winona Disallow: / User-agent: whatUseek_winona/3.0 Disallow: / User-agent: WhizBang! Lab Disallow: / User-agent: WhoWhere Robot Disallow: / User-agent: Wild Ferret Web Hopper #1, #2, #3 Disallow: / User-agent: Wildsoft Surfer Disallow: / User-agent: Willow Internet Crawler by Twotrees V2.1 Disallow: / User-agent: Wind.it client user-agent ? Disallow: / User-agent: Windows ME, BTOpenworld Internet services Disallow: / User-agent: Windows ME, Internet Explorer URL check Disallow: / User-agent: WinME, Opera 5.x Disallow: / User-agent: WinXP Disallow: / User-agent: Wired Digital Disallow: / User-agent: WISEbot/1.0 (WISEbot@koreawisenut.com; http://wisebot.koreawisenut.com) Disallow: / User-agent: WiseWire-Spider2 Disallow: / User-agent: wobot/1.0 from 206.214.202.45 Disallow: / User-agent: WWW Collector Disallow: / User-agent: www.arianna.it Disallow: / User-agent: www.freeloader.com. Disallow: / User-agent: www2000.ogsm.vanderbilt.edu Disallow: / User-agent: WWWC Ver 0.2.5 Disallow: / User-agent: WWWOFFLE/2.x Disallow: / User-agent: Xenu Link Sleuth 1.xx Disallow: / User-agent: Xenu's Link Sleuth 1.1a Disallow: / User-agent: XGET Disallow: / User-agent: xpc-mii.net HTTP server message Disallow: / User-agent: XYLEME Robot Disallow: / User-agent: xyro_(xcrawler@cosmos.inria.fr) Disallow: / User-agent: ZBot/1.00 (icaulfield@zeus.com) Disallow: / User-agent: Zeus 1680 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus 37389 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus 86122 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus ThemeSite Viewer Webster Pro V2.9 Win32 Disallow: / User-agent: Zippy v2.0 - Zippyfinder.com Disallow: / User-agent: ZyBorg/1.0 (ZyBorg@WISEnut.com; http://www.WISEnut.com) Disallow: /