Crawl, render, index: what search engines do with a Shopify store

When someone says "Google is not picking up our new products", they have described a symptom that could sit in any of three places, and the fixes do not overlap. Crawling is discovery and fetching: does the crawler know the URL exists, is it allowed to request it, and did the request succeed. Rendering is turning the fetched response into a page with content in it, which for anything JavaScript-dependent means a second pass with a browser engine. Indexing is the editorial decision: given everything that was rendered, which URLs are worth storing and serving.

The reason this matters on Shopify specifically is that the platform is unusually good at the first stage and offers you unusually many ways to make a mess of the third. You will rarely fix a Shopify indexing problem by doing more of what helps crawling, and the common interventions — submitting the sitemap again, adding noindex to things — routinely target the wrong stage.

So the useful diagnostic question is not "is it indexed" but "which stage did it stop at". Search Console answers this directly if you read the coverage report rather than the summary number.

Every Shopify store generates a sitemap automatically. The documentation describes it as containing "links to all your products, primary product image, pages, collections, and blog posts", and the file you submit is an index — Shopify's own words are that the generated sitemap files "link to separate sitemaps for your products, collections, blogs, and webpages", updated as the store changes. There is nothing to build, and no plausible reason to install an app that builds a second one.

Two caveats are worth knowing because both catch teams out. The first is private mode: Shopify is explicit that "a sitemap reader (such as Google) can't access it if your website is set to private mode", which means a password-protected store is invisible on purpose. If a site has been live for a fortnight and nothing is indexed, check the storefront password before anything else. The second is international domains — sitemaps for those are documented as available on the Basic, Shopify, Advanced and Plus plans, so a store selling across several domains on a lower tier is discovering less than it thinks.

robots.txt is the other half of discovery, and Shopify generates one by default. The documentation says the default "works for most shops, so this template isn't included in any themes by default", and that the default rules are "mirrored through the Liquid robots object", which robots.txt.liquid uses to output them. If you need your own rules, add the template and append — do not paste static text over it. Shopify's stated reason is worth quoting because it is the whole argument: "The default rules are updated regularly to ensure that SEO best practices are always applied." A hand-written robots.txt is a snapshot of one afternoon's thinking that nobody will revisit.

The practical crawl work left to you, then, is small: make sure the store is public, make sure your internal linking actually reaches deep pages rather than relying on the sitemap alone, and do not disallow anything you also want canonicalised. A blocked URL cannot tell a crawler where its canonical version lives.

If your theme uses the Section Rendering API, three documented details shape how you should build around it. It renders up to five sections per request. It does not let you specify section setting values — existing or default values apply. And a failed section comes back as null inside an otherwise successful response, so a request can return HTTP 200 with nothing useful in it.

That last one is the interesting one for search. A section that silently fails renders as an empty region on a page that reports success; nothing in your monitoring notices, and a crawler that does render the page sees a collection with no products in it. Null-check the response and fail loudly.

The general rule is simpler than the mechanics: whatever you need indexed should be in the server-rendered response. Use the API to make a page feel fast to a shopper, not to deliver content that has to be read. The test takes a minute — request the page with JavaScript disabled, or fetch it with curl, and read what comes back. If the products are not in it, the crawler is doing extra work to find them, and sometimes it will not.

This is the stage that matters, and the problem is nearly always the same one: the store publishes many URLs that are the same page. Shopify does not do this to spite you — each source is a feature somebody wanted.

The variant parameter. The Liquid documentation gives product.url as a relative URL and its own example includes a variant parameter: /products/gorgeous-wooden-computer?variant=1234567890. It also documents that when products appear in search results or filtered collections, the URL carries the variant parameter of the most relevant variant, and that product recommendations append tracking parameters instead. So a product with twelve variants has at least thirteen reachable URLs before anyone has filtered anything, and your internal links point at several of them.

Filter combinations. Storefront filtering multiplies rather than adds, and a collection with five multi-select filters reaches six figures of URLs on its own. That deserves its own treatment and has it.

Collection-scoped product paths, tag pages, and app parameters. Every one of these produces a URL that renders the same content with a different address.

The lever is the canonical tag. Liquid exposes canonical_url as a global object, documented simply as "the canonical URL for the current page", and your theme decides what it does with it. That vagueness is the point to take seriously: do not assume what your store emits, open a filtered collection URL and a ?variant= product URL in your browser and read the source. Whatever is in the tag is your actual baseline, regardless of what the theme's README claims.

And be clear about which tool does which job. noindex keeps a URL out of results but does not save a crawl, because the crawler has to fetch the page to read the directive. robots.txt saves the crawl but hides the canonical. Canonicals consolidate. Three tools, three jobs, and using the wrong one is why most faceted-navigation fixes change nothing.

Everything above is defensive. It stops you competing with yourself; it does not earn a position. What earns positions is pages that answer a query better than the alternatives, which on a store means real collection pages for the category phrases people actually search — with their own h1, their own opening paragraph, their own buying guidance and their own internal links.

The reason most teams have three of these rather than thirty is production cost, not strategy. If every landing page needs a developer, the eleventh one never gets built, and the backlog quietly becomes an argument about priorities. A [section library](/services/build/shopify-section-library) — composable blocks a merchandiser arranges without touching code — turns that from a sprint into an afternoon, which is the only intervention we have seen reliably change how many good pages a store has.

Put crudely: the crawl and render stages are hygiene, and hygiene has a ceiling. The index stage is where you decide what you are asking to be ranked, and that is a content decision with a tooling dependency.

• Shopify section libraries for content-heavy brands
• Faceted navigation without wrecking your crawl budget
• Collection pages are your best SEO asset