How to stop search engines from crawling the whole website?

Question

I want to stop search engines from crawling my whole website.

I have a web application for members of a company to use. This is hosted on a web server so that the emplo

Answer 1

If security is your concern, and locking down to IP addresses isn't viable, you should look into requiring your users to authenticate in someway to access your site.

That would mean that anyone (google, bot, person-who-stumbled-upon-a-link) who isn't authenticated, wouldn't be able to access your pages.

You could bake it into your website itself, or use HTTP Basic Authentication.

https://www.httpwatch.com/httpgallery/authentication/