Canadian Charity Annual Report Scraper

Built a layered discovery workflow that walks each charity's site, moving from the homepage to About Us and on to Accountability, Resources and Publications, scanning every page visited for either a direct report PDF link or a dedicated 'reports' landing page, so no single fixed URL pattern is assumed.

Designed a ranked, keyword-driven matching system (centralized in config.py) that checks URL, link text, and up to 500 characters of surrounding DOM context against curated keyword lists, entirely regex and BeautifulSoup, with zero LLM or paid API calls involved.

Implemented latest-year resolution that parses years (including ranges like '2025-2026') out of every candidate PDF URL and automatically keeps the most recent report, while filtering out financial statements flagged by filename pattern so they're never mis-filed as annual reports.

Engineered resilient HTTP handling, adding automatic retry with a spoofed browser User-Agent and extended timeout for sites that block default Python request headers, plus chunked-encoding-safe file downloads with backoff, so one unresponsive site never stalls the batch.

Ran the pipeline end-to-end across 1,700+ Canadian charities, writing Found / Not Found / Downloading Error back into a CSV ledger and organizing every downloaded PDF into a Business-Number/year-keyed folder structure, with idempotent re-runs that skip already-resolved charities.

Instrumented structured logging (separate info and error logs) capturing every request, discovery decision, download, and retry, keeping a 1,700+ site unattended batch run fully debuggable.