Robots.txt Builder

Allow WordPress admin AJAX

Allows WordPress background AJAX requests that some plugins and themes rely on. Recommended for most WordPress websites.

Allow: /wp-admin/admin-ajax.php

Block /wp-includes/

Prevents search engines from crawling WordPress core system files that usually do not provide SEO value.

Disallow: /wp-includes/

Block internal search pages

Prevents search engines from indexing internal site search results pages, which are often considered low-quality or duplicate content.

Disallow: /search/ Disallow: /*?s=

Block tag archive pages

Helps reduce duplicate or thin-content archive pages commonly generated by blogging systems like WordPress.

Disallow: /tag/

Block query string URLs

Prevents search engines from crawling URLs with query parameters. Useful for reducing duplicate content, but can accidentally block important pages if used incorrectly.

Example: some online stores use query strings for product filtering like:

/products?category=shirts

Blocking all query string URLs could prevent search engines from discovering those filtered product pages entirely.

Disallow: /*?