NodeJS robots.txt parser with support for wildcard (*) matching.
-
Updated
Aug 7, 2026 - JavaScript
#
rfc-9309
Here are 6 public repositories matching this topic...
A parser for robots.txt with support for wildcards. See also RFC 9309.
parser erlang parsing crawling robots-txt parsing-library robots-parser erlang-library robots-exclusion-standard rfc-9309
-
Updated
Nov 21, 2023 - Erlang
Validates whether GPTBot, ClaudeBot, PerplexityBot and other LLM crawlers are allowed by a robots.txt file.
python geo robots-txt rfc-9309 gptbot generative-engine-optimization perplexitybot claudebot crawler-policy compliance-geo llm-crawlers
-
Updated
May 10, 2026 - Python
Parse robots.txt and sitemaps using dotnet
-
Updated
Mar 6, 2026 - C#
An RFC 9309 robots.txt parser and matcher for Elixir, with Google-compatible matching behavior and no runtime dependencies.
-
Updated
Jul 14, 2026 - Elixir
An RFC 9309-conformant robots.txt parser for Python (stdlib's urllib.robotparser resolves rule conflicts by file order, not specificity)
-
Updated
Jul 4, 2026 - Python
Improve this page
Add a description, image, and links to the rfc-9309 topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the rfc-9309 topic, visit your repo's landing page and select "manage topics."