User-agent: * Disallow: /wusage/ Disallow: /mt-static/ Disallow: /cgi-bin/ Disallow: /pictures/ Disallow: /img/ Disallow: /includes/ Disallow: /alireaussi/articles/img/ Disallow: /manager/ Disallow: /meta/ Disallow: /css/ # http://www.webmasterworld.com/robots.txt has a long list of active robots you might want to block. # Some of these (and many others) ignore robots.txt, and are forcibly blocked in .htaccess. User-agent: TurnitinBot User-agent: NPbot User-agent: psbot User-agent: baiduspider User-agent: larbin User-agent: EmailSiphon User-agent: ia_archiver User-agent: NationalDirectory User-agent: LNSpiderguy User-agent: Teleport User-agent: MIIxpc User-agent: Website Quester User-agent: WebStripper User-agent: WebSauger User-agent: WebCopier User-agent: NetAnts User-agent: Mister PiX User-agent: asterias User-agent: Harvest User-agent: Bullseye User-agent: Crescent User-agent: CherryPicker User-agent: WebBandit User-agent: Microsoft URL Control User-agent: lwp-trivial User-agent: LinkWalker User-agent: cosmos User-agent: Offline Explorer User-agent: UrlDispatcher User-agent: WebEnhancer User-agent: Openfind User-agent: Openbot User-agent: Zeus User-agent: Webster Pro User-agent: MSIECrawler User-agent: sitecheck.internetseer.com User-agent: pompos User-agent: curl User-agent: Wget Disallow: /