Who is this guide for?
If you own, manage, monetize, or promote online content via Google Search, this guide is meant for you. You might be the owner of a growing and thriving business, the webmaster of a dozen sites, the SEO specialist in a Web agency or a DIY SEO ninja passionate about the mechanics of Search : this guide is meant for you. If you're interested in having a complete overview of the basics of SEO according to our best practices, you are indeed in the right place. This guide won't provide any secrets that'll automatically rank your site first in Google (sorry!), but following the best practices outlined below will hopefully make it easier for search engines to crawl, index and understand your content.
Search engine optimization (SEO) is often about making small modifications to parts of your website. When viewed individually, these changes might seem like incremental improvements, but when combined with other optimizations, they could have a noticeable impact on your site's user experience and performance in organic search results. You're likely already familiar with many of the topics in this guide, because they're essential ingredients for any web page, but you may not be making the most out of them.
You should build a website to benefit your users, and any optimization should be geared toward making the user experience better. One of those users is a search engine, which helps other users discover your content. Search Engine Optimization is about helping search engines understand and present content. Your site may be smaller or larger than our example site and offer vastly different content, but the optimization topics we discuss below should apply to sites of all sizes and types.
- Help Google find your content
- Tell Google which pages shouldn't be crawled
- Help Google (and users) understand your content
- Manage your appearance in Google Search results
- Organize your site hierarchy
- Optimize your content
- Optimize your images
- Make your site mobile-friendly
- Promote your website
- Analyze your search performance and user behavior
- Appendix: Link URLs used in this paper
Tell Google which pages shouldn't be crawled
Best Practices
For non-sensitive information, block unwanted crawling by using robots.txt
Ex: https://www.whatsapp.com/robots.txt
A "robots.txt" file tells search engines whether they can access and therefore crawl parts of your site. This file, which must be named "robots.txt", is placed in the root directory of your site. It is possible that pages blocked by robots.txt can still be crawled, so for sensitive pages you should use a more secure method.
You may not want certain pages of your site crawled because they might not be useful to users if found in a search engine's search results. If you do want to prevent search engines from crawling your pages, Google Search Console has a friendly robots.txt generator to help you create this file. Note that if your site uses subdomains and you wish to have certain pages not crawled on a particular subdomain, you'll have to create a separate robots.txt file for that subdomain. For more information on robots.txt, we suggest this Webmaster Help Center guide on using robots.txt files13.
Read about several other ways to prevent content from appearing in search results.14
Avoid:
Don't let your internal search result pages be crawled by Google. Users dislike clicking a search engine result only to land on another search result page on your site.
Allowing URLs created as a result of proxy services to be crawled.
For sensitive information, use more secure methods
Robots.txt is not an appropriate or effective way of blocking sensitive or confidential material. It only instructs well-behaved crawlers that the pages are not for them, but it does not prevent your server from delivering those pages to a browser that requests them. One reason is that search engines could still reference the URLs you block (showing just the URL, no title or snippet) if there happen to be links to those URLs somewhere on the Internet (like referrer logs). Also, non-compliant or rogue search engines that don't acknowledge the Robots Exclusion Standard could disobey the instructions of your robots.txt. Finally, a curious user could examine the directories or subdirectories in your robots.txt file and guess the URL of the content that you don't want seen.
In these cases, use the noindex tag if you just want the page not to appear in Google, but don't mind if any user with a link can reach the page. For real security, though, you should use proper authorization methods, like requiring a user password, or taking the page off your site entirely.
Help Google (and users) understand your content
Let Google see your page the same way a user does
When Googlebot crawls a page, it should see the page the same way an average user does15. For optimal rendering and indexing, always allow Googlebot access to the JavaScript, CSS, and image files used by your website. If your site's robots.txt file disallows crawling of these assets, it directly harms how well our algorithms render and index your content. This can result in suboptimal rankings.
Recommended action:
Use the URL Inspection tool16. It will allow you to see exactly how Googlebot sees and renders your content, and it will help you identify and fix a number of indexing issues on your site.