SEO and GEO
Your content is good, but Google does not show it
Usually because Google never reaches or processes the page cleanly in the first place. Responsible for that are different crawlers with their own tokens: Googlebot for smartphone and desktop, plus Googlebot-Image, Googlebot-Video, Googlebot-News, Google StoreBot, Google-InspectionTool and GoogleOther g-crawler. Lock one of them out by accident and you lose exactly that channel. The second common cause is signals that contradict each other: a page meant to be indexed and excluded at the same time.
What we do: We collect the whole inventory, check by template instead of by page, verify crawler access and resolve conflicting signals; Google processes your content completely, and one error no longer costs ten thousand pages.
By Raymond Eiber, Organic Search Channel Lead
Last updated on
What it is about and when it becomes relevant
What this is about
Technical SEO makes sure Google can find, load, process and classify. Google lists the common crawlers with their tokens: Googlebot for smartphone and desktop, Googlebot-Image, Googlebot-Video, Googlebot-News, Google StoreBot, Google-InspectionTool and GoogleOther including variants g-crawler. Every block hits exactly the area that crawler is responsible for.
When it becomes relevant
After every relaunch, with every move to a new system, and whenever pages are listed as not indexed in Search Console although they look reachable. Also with large inventories, because small errors there act at scale.
Why breadth matters
A single error on one page costs one page. The same error in a template costs ten thousand. Which is why checking here happens by pattern, not by page: which template, which page type, which section.
Common mistakes and how you spot them
Crawlers locked out across the board
How you spot itThe robots.txt carries a rule that hits more than intended. Images or videos vanish from search while the text pages remain g-crawler.
Contradicting signals
How you spot itA page is blocked in robots.txt and carries a noindex at the same time. Google then cannot read the noindex, and the page stays in the index.
Content only there after script execution
How you spot itThe source code contains almost nothing; the content only appears in the browser. In Search Console, exactly this content is missing from the rendered version.
Redirect chains after the relaunch
How you spot itOld addresses point to the new one across several hops. Visibility gets lost along the way, and the report shows sinking impressions for pages that still exist.
How we go about it
Check by pattern, not by page
We collect the inventory completely and group by template and page type. That turns thousands of alerts into a handful of causes.
Cross-check crawler access
We look at which tokens actually access the site and which are locked out g-crawler, instead of only reading the rules file.
Resolve contradictions
Blocks, redirects, canonicals and noindex belong read together. One signal has to win, and deliberately so.
Measure after the intervention
We track indexing and impressions per page type, so the effect stays attributable.
Further reading
This topic belongs to SEO.
Frequently asked questions
Which Google crawlers exist?
Google lists, among others, Googlebot for smartphone and desktop, Googlebot-Image, Googlebot-Video, Googlebot-News, Google StoreBot, Google-InspectionTool and GoogleOther including variants g-crawler.
Why does a blocked page stay in the index?
Because a block in robots.txt prevents reading. Google then cannot see a noindex on that page. Whoever wants a page out of the index has to leave it readable.
Is load time a ranking factor?
It is part of the user experience and acts mostly through how many visitors stay. Before fine-tuning, though, the question always worth asking is whether Google processes the content completely at all.
How often should I check technically?
After every relaunch and every larger system change, plus continuously via automated monitoring. The expensive errors happen during change, not in normal operation.
Is this exactly where you are stuck?
Write to us about where it is stuck. Raymond Eiber and the team will look at what is concretely behind it in your case, and tell you where the biggest lever sits.
Sources
Every statement about ad products comes from the official documentation of the respective platform, with the date we retrieved it. Products change; if something looks outdated to you, write to us and we will check it.
- g-crawlerGoogle Search Central, Gängige Crawlerretrieved on 21.9.2026
Content and images on this page may have been created with AI support.