Tester voor robots.txt
Test robots.txt rules online for user-agent matching, allow and disallow paths, and crawl directive limits. Nederlands locale.
Make HTML structure easier to inspect
This tool reads robots.txt text and helps evaluate which paths a selected crawler user agent appears allowed or disallowed...
Formatting is not HTML validation
A robots.txt file contains groups with user-agent selectors and rules such as Allow and Disallow, plus optional sitemap an...
How to format HTML
- Enter or paste robots.txt, choose a user agent, test representative paths, and review the matching group and rule. Check t...
- Enter or paste robots.txt, choose a user agent, test representative paths, and review the matching group and rule. Check t...
Formatting method
- Read markup structure
The tester parses groups, matches user-agent tokens, evaluates path rules, and explains the selected result. It may model ...
- Apply indentation
The tester parses groups, matches user-agent tokens, evaluates path rules, and explains the selected result. It may model ...
- Preserve content
The tester parses groups, matches user-agent tokens, evaluates path rules, and explains the selected result. It may model ...
Exact parser recovery, optional-tag handling, whitespace policy, embedded-language behavior, and error wording should be c...
Read the formatted result
A useful result shows selected user agent, tested URL path, matching rules, and any assumptions. Check specificity, group ...
Veelgemaakte fouten
Common mistakes include blocking CSS or JavaScript needed for rendering, assuming robots.txt protects private files, forge...
Common mistakes include blocking CSS or JavaScript needed for rendering, assuming robots.txt protects private files, forge...
Grenzen en bijzondere gevallen
The tester cannot enforce crawler behavior, remove indexed content, validate server access control, or guarantee every bot...
The tester cannot enforce crawler behavior, remove indexed content, validate server access control, or guarantee every bot...
Examples
robots.txt Tester online — 1
What does robots.txt control?
Enter or paste robots.txt, choose a user agent, test representative paths, and review the matching group and rule. Check t...
A useful result shows selected user agent, tested URL path, matching rules, and any assumptions. Check specificity, group ...
robots.txt Tester online — 2
Is it a security feature?
Enter or paste robots.txt, choose a user agent, test representative paths, and review the matching group and rule. Check t...
A useful result shows selected user agent, tested URL path, matching rules, and any assumptions. Check specificity, group ...
robots.txt Tester online — 3
How is the user agent selected?
Enter or paste robots.txt, choose a user agent, test representative paths, and review the matching group and rule. Check t...
A useful result shows selected user agent, tested URL path, matching rules, and any assumptions. Check specificity, group ...
robots.txt Tester online — 4
Does Disallow remove pages from search?
Enter or paste robots.txt, choose a user agent, test representative paths, and review the matching group and rule. Check t...
A useful result shows selected user agent, tested URL path, matching rules, and any assumptions. Check specificity, group ...
robots.txt Tester online — 5
What is Allow?
Enter or paste robots.txt, choose a user agent, test representative paths, and review the matching group and rule. Check t...
A useful result shows selected user agent, tested URL path, matching rules, and any assumptions. Check specificity, group ...
robots.txt Tester online — 6
Are wildcards universal?
Enter or paste robots.txt, choose a user agent, test representative paths, and review the matching group and rule. Check t...
A useful result shows selected user agent, tested URL path, matching rules, and any assumptions. Check specificity, group ...
Frequently asked questions
What does robots.txt control?
robots.txt gives crawl guidance to compliant crawlers. Use authentication and server controls for access, and manage index...
Is it a security feature?
robots.txt gives crawl guidance to compliant crawlers. Use authentication and server controls for access, and manage index...
How is the user agent selected?
robots.txt gives crawl guidance to compliant crawlers. Use authentication and server controls for access, and manage index...
Does Disallow remove pages from search?
robots.txt gives crawl guidance to compliant crawlers. Use authentication and server controls for access, and manage index...
What is Allow?
robots.txt gives crawl guidance to compliant crawlers. Use authentication and server controls for access, and manage index...
Are wildcards universal?
robots.txt gives crawl guidance to compliant crawlers. Use authentication and server controls for access, and manage index...
Does crawl-delay work everywhere?
robots.txt gives crawl guidance to compliant crawlers. Use authentication and server controls for access, and manage index...
Where should robots.txt live?
robots.txt gives crawl guidance to compliant crawlers. Use authentication and server controls for access, and manage index...