In previous posts, we've already covered scraping publicly available data from Instagram and Airbnb. This time, we'll show you how to scrape walmart.com to collect data about products by category. Walmart is a rich source of data containing:
- product details (including category and description) 📝
- price 💰
- features (e.g. nutrition facts for food) 🥦
- availability 🛒
- reviews ⭐️
As always, we share the code in GitHub repository to let you play with it and apply to your use case. To be able to run it and actually scrape the data, you need Scraping Fish API key which you can get here: Scraping Fish Requests Packs. A starter pack of 1,000 API requests costs only $2. Without Scraping Fish API you are likely to see captcha instead of useful product information.
Scraping use case
As a 🏃♂️ running example, we'll identify products from 🍔 food categories and scrape nutrition facts data for them. Based on collected data for over 15,000 products, we'll find out:
- What is the share of products having sugar as the main nutrient?
- Is there any relation between product rating and its nutrients?