Technical SEO Basics: Crawling, Indexing, Speed & Mobile

Updated Oct 26, 2026 · 2 min read

Technical SEO is making sure search engines can find, read and trust your site — before content quality even enters the picture. Most of it is unglamorous plumbing, and most of it is one-time setup.

Crawlability: robots.txt and internal links

A crawler discovers pages by following links. Every important page should be reachable within a few clicks of the homepage, and your robots.txt should block only what should stay out (admin areas, internal search results). Test rules before deploying them — one wrong Disallow can hide your whole site — with our Robots.txt Tester, and generate a clean file with Robots.txt Generator.

Sitemaps: the map you hand Google

An XML sitemap lists your pages for crawlers, with last-modified dates. It does not guarantee indexing, but it speeds discovery — especially for new sites with few backlinks. Generate one with our XML Sitemap Generator and submit it in Google Search Console. Keep it updated automatically if you can; a stale sitemap is worse than none.

HTTPS, speed and Core Web Vitals

HTTPS is a baseline ranking signal and a browser trust requirement — no modern site should serve HTTP. Speed matters through Core Web Vitals: Largest Contentful Paint (main content visible, aim under 2.5s), Interaction to Next Paint (responsiveness), and Cumulative Layout Shift (no jumping content). Compress images, avoid render-blocking scripts, and test with PageSpeed Insights — then fix the biggest offender first.

Mobile-first and structure

Google predominantly uses the mobile version of pages for indexing and ranking. Your mobile page must contain the same content, headings and structured data as desktop — a stripped-down mobile page ranks on stripped-down content. Add hreflang tags for multi-language sites (our Hreflang Tag Generator builds them) and canonical tags where duplicates exist.

JavaScript SEO in one paragraph

Google renders JavaScript pages, but rendering happens later than plain HTML crawling — content that only appears after scripts run can be indexed late or missed. The safe rule: critical content (headings, main text, links) should exist in the HTML itself, not only in JavaScript. Test with Search Console's URL inspection tool, which shows the rendered page Google actually sees — if your key content is missing there, it is missing for Google too.

Duplicate content and canonicals

Duplicates usually come from URLs, not copy-paste: www vs non-www, trailing slashes, and tracking parameters (?utm_source=...) each create a second address for the same page. Pick one canonical version per page, point the canonical tag at it, and link internally to that version consistently. Without canonicals, ranking signals split across duplicates and none of them rank as well as one would.

Frequently asked questions

How do I know if Google indexed my pages?

Google Search Console → Pages report, or search site:example.com to see indexed pages (approximate). Unindexed pages usually suffer from crawl blocks, thin content, or no internal links.

What is the biggest technical SEO mistake?

Accidentally blocking crawlers — a stray Disallow in robots.txt, a noindex tag left from staging, or a site launched behind a login. Check Search Console's coverage report after every launch.

Do I need a developer for technical SEO?

For the basics here, no. For large sites (JavaScript rendering, faceted navigation, international setups), yes — that is where technical SEO becomes a specialty.

Is a sitemap enough for indexing?

No — a sitemap speeds discovery, it does not guarantee indexing. Indexing also needs crawlable links, real content quality, and no blocks (robots.txt, noindex). A sitemap with thin pages behind it just helps Google find thin pages faster.

How fast should pages load?

Aim for Largest Contentful Paint under 2.5 seconds — that is Google's 'good' threshold. But speed is one signal among many: a fast thin page still loses to a slower thorough one. Fix the biggest offender first (usually images), then move on to content.