Post #2686061

@tdp_org@mastodon.social This is interesting information and certainly something I will consider making use of myself. All legitimate crawlers should make it easy to verify the authenticity of the requests they make. I know that for Googlebot the recommended way to verify the authenticity of the request is as follows: Do a reverse lookup of the IP address.Verify that the resulting name is under the googlebot.com domain.Do a forward lookup of the name to verify you get the original IP address back. I don’t know if similar steps have been published for other legitimate crawlers.