Robots.txt is text a file which contains a set of commands and tells search engines crawler or bot which files to index or not to index.
Get and save the most relevant links from around the web. Tailored for you.
Or continue with Twitter, Facebook, Google
Already a member? Sign in