There are a lot of self-declared "optimized" robots.txt files for WordPress around the web, but most of them don't allow full access to most Googlebots that scans our website for various reasons, such as: mobile browsers compatibility, PageSpeed insights, tag/structured data analysis, & more. Here's a list of all of them as of October 2016, including the most recent ones.
If you want to be sure that your robots.txt isn't blocking any of these, you can test your website URLs with the following tools:
- Mobile-Friendly Test: https://www.google.com/webmasters/tools/mobile-friendly/
- PageSpeed Insights: https://developers.google.com/speed/pagespeed/insights/
- Fetch as Google (requires a registered property in Google Search Console): https://support.google.com/webmasters/answer/6066468?hl=en
If you're unhappy with the results, I strongly suggest to adopt something similar to the following:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
User-agent: * Allow: / Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content Disallow: /e/ Disallow: /show-error-* Disallow: /xmlrpc.php Disallow: /trackback/ Disallow: /comment-page- Allow: /wp-content/uploads/ User-agent: Mediapartners-Google Allow: / User-agent: Adsbot-Google Allow: / User-agent: AdsBot-Google-Mobile-Apps Allow: / User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Mobile Allow: / User-agent: Googlebot-News Allow: / User-agent: Googlebot-Video Allow: / Sitemap: http://YOUR_SITEMAP_URL |
IMPORTANT: Be sure to replace YOUR_SITEMAP_URL with your actual sitemap URL: if you're using WordPress+Jetpack, it should be something like www.yoursite.com/sitemap.xml.
That's it for now: happy SEO!
