Directives/Middleware - http.nobots - 《Caddy v1.0 Document》

http.nobots Plugin
- Examples

http.nobots Plugin

This feature does not come with Caddy by default. To get it, select the http.nobots plugin when you download Caddy.

Nobots protect your website against web crawlers and bots.

Examples

Ban bots by user agent strings

nobots "bomb.gz" { "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "Googlebot/2.1 (+http://www.google.com/bot.html)" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "Googlebot-News" "Googlebot-Image/1.0" "Googlebot-Video/1.0" "compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html" "Mediapartners-Google" "AdsBot-Google (+http://www.google.com/adsbot.html)" "AdsBot-Google-Mobile-Apps" "APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)" }

It will send the bomb.gz to the Google bots listed inside the directive block.

Ban bots by using regular expresions

nobots "bomb.gz" { regexp "bingbot" }

It will send the bomb.gz to all user agents that contains bingbot.

Mix of strings and regular expresions

nobots "bomb.gz" { "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)" regexp "bingbot" }

It will send the bomb.gz to all user agents that contains bingbot and the one defined on the string as well.

Unproctected URI

nobots "bomb.gz" { "Googlebot" public "^/public" public "^/[a-z]{,5}/public" }

It will send the bomb.gz to all user agents that match Googlebot and the URI is not defined in the public keyword.