Python Web Scraping — Examples

Runnable, tested Python web scraping examples — the companion code for python-web-scraping.com.

Every example is a small, self-contained function with a test. Each module maps to a guide on the site, so you can read the explanation there and run the code here.

Quick start

git clone https://github.com/python-web-scraping-com/examples.git
cd examples
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e ".[dev]"

Run any example directly:

python -m pws_examples.parsing
python -m pws_examples.storage
python -m pws_examples.async_scrape   # this one makes real HTTP requests

Run the tests and linter:

pytest
ruff check .

What's inside

Module	What it shows	Guide
pws_examples/parsing.py	Parse product cards and HTML tables with BeautifulSoup	Parsing HTML with BeautifulSoup
pws_examples/extraction.py	Extract emails and phone numbers with regex	Extracting Data with Regular Expressions
pws_examples/http_client.py	A requests session with retries and backoff	Understanding HTTP Requests and Responses
pws_examples/storage.py	Validate with Pydantic, store in SQLite, de-duplicate	Storing and Exporting Scraped Data
pws_examples/async_scrape.py	Concurrent fetching with asyncio + HTTPX and a semaphore	Asynchronous Scraping with asyncio and HTTPX
pws_examples/pagination.py	Collect all items across next-link, offset/limit and cursor pagination	Handling Pagination and Infinite Scroll
pws_examples/sessions.py	Log in via a form, persist cookies, reuse them across requests	Managing Cookies and Sessions
pws_examples/json_ld.py	Extract JSON-LD / schema.org Product data from ld+json blocks	Extracting JSON-LD and Structured Data
pws_examples/retry.py	Retry 429/5xx with exponential backoff, jitter and Retry-After via tenacity	Retrying Failed Requests with tenacity
pws_examples/user_agents.py	Rotate user agents and build realistic browser-like headers	How to Rotate User Agents in Python
pws_examples/robots.py	Parse robots.txt: path permissions, crawl-delay and sitemaps	How to Scrape a Static Website Without Getting Blocked
pws_examples/export.py	Export scraped rows to CSV and JSON Lines with the stdlib	Storing and Exporting Scraped Data

The tests use local HTML fixtures and httpx.MockTransport, so the suite is deterministic and runs offline — no live websites are hit in CI.

Scrape responsibly

These examples are for learning. When scraping real sites, respect robots.txt, rate-limit your requests, identify your client honestly, and follow each site's terms of service.

Contributing

Contributions are welcome. Please keep each example small and focused, add a test for it, and make sure pytest and ruff check . pass before opening a PR.

License

MIT © Python Web Scraping