Start with a tight SKU map
Pick five to ten rival stores and your top sellers. Map each item to a match on color, size, and pack count. Keep it strict, since bad matches ruin the feed.
Record more than list price. Capture ship cost, promo text, and stock state. A rival with a low price and no stock still gives you room to win.
Scrape with care so you do not get blocked
Most blocks hit when your bot looks like a flood. Cap your request rate and spread runs across the day. Cache pages for a set time so repeat pulls do not slam the same site.
Use a proxy pool when you need geo match or you face hard limits. A residential pool often works best for retail sites that flag data center IPs. Many teams start with Byteful.
Keep your scraper honest. Set a real user agent, handle cookies, and follow normal page flows. Retry only a few times, and log every fail with a clear reason.
Normalize the data so it drives actions
Convert all prices to one unit and one tax view. Strip symbols, parse ranges, and store both the raw text and the clean number. Save a time stamp for each pull.
Build one “landed cost” field for rivals. Add ship, and note ship speed when you can. This one field often beats a basic price match rule.