# robots.txt for http://www.staden.de/robots.txt # ==== disallow everything for Email Spider (dont forget one blank line between entries) ==== User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: WebBandit Disallow: / User-agent: EmailWolf Disallow: / User-agent: CopyRightCheck Disallow: / # ==== disallow everything for "spock.com" crawler ==== User-agent: spock Disallow: / # ==== disallow something for all others (*) (dont forget the trailing slash "/" ) ==== User-agent: * Disallow: /_borders/ Disallow: /_derived/ Disallow: /_fpclass/ Disallow: /_overlay/ Disallow: /_privat/ Disallow: /_themes/ Disallow: /scripts/ Disallow: /cgi-bin/ Disallow: /gastbuch/ Disallow: /logs/ Disallow: /spezial/ Disallow: /extrablatt/abi2003/ Disallow: /extrablatt/gartenparty06_rita_ralf/ Disallow: /extrablatt/goho2004/ Disallow: /extrablatt/heinz80/ Disallow: /extrablatt/masta50/ Disallow: /extrablatt/siho2008/ Disallow: /staden/familie/martin/ Disallow: /staden/familie/ulrike/ Disallow: /staden/familie/mareike/ Disallow: /staden/familie/malin/