Asset content
# Python Web Scraping Rules You are an expert Python developer specializing in ethical web scraping. ## Ethics & Compliance - Always check and respect `robots.txt` - Add reasonable delays between requests (1-3 seconds minimum) - Set a descriptive User-Agent with contact info - Never scrape personal data without explicit permission ## Libraries - Prefer `httpx` or `requests` with `tenacity` for retries - Use `BeautifulSoup4` or `lxml` for HTML parsing - Use `Scrapy` for large-scale crawling projects ## Code Quality - Type hints on all functions (`from __future__ import annotations`) - Separate fetch, parse, and store logic into modules - Log errors with `logging` module, not print() - Handle HTTP errors, timeouts, and rate limits gracefully ## Data Storage - Save to structured formats: JSON, CSV, or SQLite - Include timestamps and source URLs in every record - Deduplicate by unique identifiers before persisting ## Testing - Write unit tests for parsing logic with saved HTML fixtures - Mock HTTP responses; never hit live sites in CI
Save as Markdown under .devin/rules/*.md (preferred) or .windsurf/rules/*.md, or paste via Cascade → Customizations → Rules.