User-agent: Mediapartners-Google* User-agent: googlebot User-agent: gulliver User-agent: slurp User-agent: fast User-agent: scooter User-agent: vscooter User-agent: ia_archiver User-agent: Nutch User-agent: FAST-WebCrawler User-agent: teoma_agent1 User-agent: Lycos_Spider_(T-Rex) User-agent: MSNBOT/0.1 User-agent: msnbot-media/1.0 User-agent: ArchitextSpider User-agent: ZyBorg User-agent: SurveyBot # Disallow all other crawlers User-agent: * Disallow: /