# Robots.txt file # For domain: http://www.comclusive.co.uk # All robots will spider the domain User-agent: * Disallow: /var/www/html/new/ Disallow: /var/www/html/billing/ Disallow: /var/www/html/old_data/