The process by which search engine software discovers and accesses webpages and their links.
Crawling is the first technical step that allows pages to be considered for indexing.
Short and sweet:
The process by which search engine software discovers and accesses webpages and their links.
Crawling is the first technical step that allows pages to be considered for indexing.
See also: Indexing, XML Sitemap, Technical SEO
Crawling is the process by which search engines send automated bots to discover pages across the web, following links from page to page to find new or updated content worth evaluating. A page has to be crawled before it can even be considered for indexing, which is why broken links, blocked pages, or poor site structure can prevent otherwise good content from ever showing up in search results. Site owners can influence crawling through tools like robots.txt and internal linking, though they can’t force a search engine to crawl anything on demand.
Googlebot performs crawling around the clock, following links from page to page across a site like Wikipedia to discover new and updated articles as they’re published.
A site owner can block this process on certain pages using a robots.txt file, telling Googlebot to skip an internal staff directory or a duplicate staging version of the site.
When a brand-new blog publishes its first post, it may take days to get discovered naturally, which is why many site owners submit the URL directly through Google Search Console to speed things along.
