Competitor SEO Monitoring Pipeline
Data / Backend Engineer · Portfolio project · Local-first PoC
A local-first pipeline for monitoring competitor content and SEO changes, using Daypass as the primary website and ResortPass as the competitor.
View source & evidenceComplete article records · paid enrichment capped below $1.
How it works
Discover
robots.txt + sitemaps + links
Crawl
Bounded Python crawler
Extract
Metadata + normalized content
Store & compare
Immutable JSONL + diffs
Enrich
Approved DataForSEO requests
Report
Dashboard + auditable XLSX
Problem
Recurring competitor SEO reviews are slow and hard to audit when assembled manually, while raw HTML comparisons create noisy alerts from navigation, timestamps, and dynamic widgets.
Solution
Built two isolated paths: a free crawler, snapshot, diff, dashboard, and workbook workflow; plus optional DataForSEO enrichment guarded by dry runs, environment-only credentials, caching, and a hard cost ceiling.
Key decisions
- Used standard-library Python for the core crawler so the collection path stays reproducible without third-party Python dependencies.
- Chose robots-aware sequential crawling for ethical, auditable rate control instead of maximum throughput.
- Compared normalized main text for meaningful changes; raw HTML hashes remain diagnostic signals only.
- Made dated JSONL snapshots immutable unless replacement is explicitly requested with --force.
- Put DataForSEO behind a trust boundary: dry-run by default, environment-only credentials, allowlisted hosts, immutable caching, and a cost check before every request.
- Never fabricates missing SEO metrics; workbook formulas reference raw-data sheets and the dashboard binds only to loopback.
Tech stack
- Python 3.11+
- JSONL / JSON
- DataForSEO API v3
- Node.js
- @oai/artifact-tool
- HTML
- CSS
- JavaScript
- unittest