seodump27

Search Console - Url not indexed due to Robot.txt but still getting crawled.

by @seodump27 (112), 3 weeks ago

My Search Console displaying Url not indexed due to Robot.txt but still getting crawled. So, How to stop crwaling for a specific url even though the website is not getting indexed?

118 Views
0 Upvotes
1 Reply
1 User
Sort replies:
binayjha
by @binayjha (4974), 3 weeks ago

The web page might be crawled before you applied no index property in robots.txt file. You can simply remove the crawled URL from Google's cache using the GSC and then it should not be crawled again as per the instruction in the robots.txt file.

Join the forum to unlock true power of SEO community

You're welcome to become part of SEO Forum community. Register for free, learn and contribute.

Log In Sign up