User-agent: * list folders robots are not allowed to index Disallow: /CGI-BIN/ Disallow: /aspnet_client Disallow: /en/home/join.shtml Disallow: /en/home/login.shtml Disallow: /en/home/what.shtml