Search engine spiders are of benefit to websites and to those conducting searches. A search engine spider visits sites, copies the information, and takes the info back to the search engine They find sites through links on other sites. If you don’t want a page to be indexed by a search engine spider for whatever reasons, you must use a robots.txt file. Use the robots.txt for pages that are spammy, or have duplicate content.