In Dec 2022, amidst high inflation, the author created a website to track supermarket prices in Greece, facing challenges like Javascript rendering. Utilizing Playwright, a web browser controller, the author automated price tracking from three supermarkets. After battling IP restrictions and failures, a combination of old laptops and cloud servers was used to run scrapers efficiently. The system, featured on AWS and Hetzner, shows high reliability with minimal costs. Optimizations such as using larger servers and minimizing data fetching enhanced performance. The total cost of scraping amounted to €4.94, with data stored on Cloudflare. Overall, the scraping pipeline proved successful in tracking supermarket price changes.
https://www.sakisv.net/2024/08/tracking-supermarket-prices-playwright/