Skip to content
#

polite-crawler

Star

Here are 2 public repositories matching this topic...

Language: All
Filter by language
All 2 HTML 1 Python 1

SagarBiswas-MultiHAT / Web_Vulnerability_Scanner-AI

Star 31

PoliteWebScanner is a learning-grade web vulnerability scanner plus AI-assisted report viewer. It crawls safely (queue-based, robots-aware, rate-limited), detects common issues (security headers, cookie flags, reflected XSS heuristics, SQL error patterns), and exports human-friendly JSON/HTML reports with optional AI help for triage and remediation

python cli flask json crawler reporting rate-limiting xss pytest robots-txt sql-injection web-security html-report connection-pooling threadpool vulnerability-scanner security-headers ai-assistant cookie-security polite-crawler
  • Updated Feb 13, 2026
  • Python

ASHHADgit87 / FlyRank-BE-Task3

Star 1

A polite web scraper for FlyRank's Backend AI Engineering internship. Implements fetch → parse → extract → clean → structure pipeline against books.toscrape.com with robots.txt compliance, rate limiting, custom User-Agent identification, structured JSON output, Jest tests, and a live-demo API on Vercel.

javascript scraper etl rest-api cheerio web-crawler rate-limiting web-scraping robots-txt node-scraper node-js jest-tests express-js data-pipeline vercel polite-crawler
  • Updated Jul 27, 2026
  • HTML

Add this topic to your repo

To associate your repository with the polite-crawler topic, visit your repo's landing page and select "manage topics."

Learn more