creating page with paid articles (paywall)
Unanswered
Giant panda posted this in #help-forum
Giant pandaOP
Hi! I would like to create a page with a paywall for paid articles. I'm already familiar with structured data to provide valuable information for web crawlers like Google. However, there's a question: How can I check if an actual crawler is a legitimate one? Users can easily change their user agent to impersonate crawlers and view the paid articles. Is there any way to verify them?
2 Replies
Giant pandaOP
Of course, I also know about the property
userAgent.isBot. However, as far as I'm aware, it only uses the user agent to assume that the visitor is a bot.Toyger
google have rDNS and ip ranges https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot
for others you need to search yourself, maybe other provider also have such things.
for others you need to search yourself, maybe other provider also have such things.