# /robots.txt file for http://www.SuperBookDeals.com # right now, we are allowing spidering # if we want to disallow it, we can uncomment the following lines. #User-Agent: Googlebot #Allow: / User-agent: Mediapartners-Google* Allow: / User-agent: * Disallow: / #Disallow: /cgi-bin #the following was really hammering us #User-agent: ia_archiver #Disallow: / #User-agent: PortalBSpider #Disallow: / #User-agent: Slurp #Crawl-delay: 30 # all robots are welcome #User-agent: * #Disallow: redirect.asp #User-agent: Googlebot #Disallow: