Client workUp to ~50K posts & ~1M comments / day
Instagram Scraping
Data Engineer (solo)
Large-scale Instagram data collection for AI sentiment analysis and competitor monitoring.
Clean, structured engagement data—delivered daily.
How it works
Source
Crawl
Selenium
Orchestrate
Airflow
Store
PostgreSQL
Analyze
Sentiment AI
Problem
Hard to obtain competitor engagement data automatically.
Solution
Mass scraping of Instagram posts & comments.
Key decisions
- Benchmarked CPU, RAM, disk, and network per scraper — part of my undergraduate thesis research.
Tech stack
- Python
- Selenium
- PostgreSQL
- Airflow