Robots.txt
Read How to use robots.txt, meta robots, and canonical tags correctly 
Guide to robots.txt, meta robots, & canonical tags
SEO practitioners sometimes encounter scenarios where their carefully planned optimization efforts backfired: pages mysteriously...
Read The keys to building a Robots.txt that works 
The keys to building a Robots.txt that works
Bots, also known as Crawlers or Spiders, are programs that “travel” across the Web...