With Google aiming to make the robots.txt exclusion protocol a standard, they proposed some changes and submitted them the other day. Now, Google updated their own developer docs around the robots.tx
Get and save the most relevant links from around the web. Tailored for you.
Or continue with Twitter, Facebook, Google
Already a member? Sign in