Scraping Fish Product Updates 2024

Mateusz Buda
Mateusz Buda
Co-Founder
Paweł Kobojek
Paweł Kobojek
Co-Founder
Scraping Fish Product Updates 2024

Greetings, web scraping enthusiasts! At Scraping Fish, we've been working tirelessly behind the scenes, and we're thrilled to announce some major enhancements to our platform. These updates are not just incremental improvements, they represent a significant leap forward in making web scraping more efficient, user-friendly, and adaptable to your unique needs. Whether you're a long-time user or considering Scraping Fish for your web scraping tasks, these updates are designed to streamline your experience and expand your capabilities.

State of Web Scraping 2023 Survey Results

Mateusz Buda
Mateusz Buda
Co-Founder
Paweł Kobojek
Paweł Kobojek
Co-Founder
Jordan Hansen
Jordan Hansen
Founder & CEO of Cobalt Intelligence
State of Web Scraping 2023 Survey Results

This survey is our attempt to understand the evolving landscape of web scraping, capturing insights from a diverse pool of developers, data scientists, and hobbyists. We've collected responses on a broad range of topics, from technical skills and tool preferences to the ethical considerations and financial aspects of web scraping. In the spirit of openness and community learning, we are excited to share the raw data containing all responses. For those interested in a deeper dive, you can access the data at this link. Our goal with this survey is to paint a detailed picture of the current state of web scraping, the prevailing trends, practices, challenges, and opportunities associated with it. The rich insights garnered from these responses should help inform and stimulate valuable discussions around the future of web scraping. Now, let's dive into the findings and see what the data has to say!

State of Web Scraping 2023 Survey

Mateusz Buda
Mateusz Buda
Co-Founder
Paweł Kobojek
Paweł Kobojek
Co-Founder
State of Web Scraping 2023 Survey

Whether you're just starting your journey in web scraping or you're a seasoned expert, your insights are crucial. The survey covers a wide range of topics, from your role and expertise level in web scraping, the tools and languages you use, to the ethics and challenges you encounter in your web scraping activities.

Scraping Google SERP with Geolocation

Mateusz Buda
Mateusz Buda
Co-Founder
Paweł Kobojek
Paweł Kobojek
Co-Founder
Scraping Google SERP with Geolocation

Google Search Engine Result Pages (SERP) 🔍 are an important source of information for many businesses. SERP includes organic search results, ads, products, places, knowledge graphs, etc. For example, by scraping Google SERP you can learn which brands have paid ads for specific keywords or analyze how your website is positioned. One of the most important factors affecting what the user can see on a keyword's SERP is geolocation 🗺. We'll show you how to scrape Google SERP for a keyword from any location, regardless of the origin of your IP address. This can be done simply by setting search URL query parameters. It's also possible to parametrize and automate scraping 🤖 using a python script we share below.

Scraping Walmart to Estimate Share of Sugar in Food

Mateusz Buda
Mateusz Buda
Co-Founder
Paweł Kobojek
Paweł Kobojek
Co-Founder
Scraping Walmart to Estimate Share of Sugar in Food

In previous posts, we've already covered scraping publicly available data from Instagram and Airbnb. This time, we'll show you how to scrape walmart.com to collect data about products by category.

Build Your Own Mobile Proxy for Web Scraping

Mateusz Buda
Mateusz Buda
Co-Founder
Paweł Kobojek
Paweł Kobojek
Co-Founder
Build Your Own Mobile Proxy for Web Scraping

In this guide, we show how you can build your own mobile proxy pool step by step. The most common use case for mobile proxies is web scraping. If you have low success rate and keep getting blocked by websites you want to scrape, this guide is for you.

How Much Money Do Indie Hackers Products Make?

Mateusz Buda
Mateusz Buda
Co-Founder
Paweł Kobojek
Paweł Kobojek
Co-Founder
How Much Money Do Indie Hackers Products Make?

Indie Hackers is a wonderful community for founders, a vast ocean of shared knowledge and a place to showcase your product journey. A lot of people are sharing their products there along with the revenue, often verified by Stripe. We analyzed the available data to check how much revenue are those products making and what are the most money making categories! At the time of our analysis, there were total of 937 products with Stripe verified revenue. The information we gathered to make this analysis are revenue and product's categories.

Fact Checking with Web Scraping #1 - Are Most Rust Jobs In Crypto?

Mateusz Buda
Mateusz Buda
Co-Founder
Paweł Kobojek
Paweł Kobojek
Co-Founder
Fact Checking with Web Scraping #1 - Are Most Rust Jobs In Crypto?

Rust is a relatively new programming language without established and highly developed job market yet. Being developer's [the most loved language](https://insights.stackoverflow.com/survey/2021#technology-most-loved-dreaded-and-wanted) though, many people would like to have a full-time job using it. There's a prevalent feeling among Rustaceans that most of the Rust jobs out there involve blockchain tech. Indeed, anyone who was ever seeking for the job in Rust surely stumbled upon a cryptocurrency job offer. Is it really true that _most_ offers are in crypto? We have scraped job ads to find the answer!

Introducing JavaScript Scenario Execution

Mateusz Buda
Mateusz Buda
Co-Founder
Paweł Kobojek
Paweł Kobojek
Co-Founder
Introducing JavaScript Scenario Execution

Today, we are pleased to introduce the much awaited feature - JavaScript scenario execution! A lot of our customers were asking for the possibility of interacting with the scraped website - clicking buttons, filling out forms, selecting <select> options etc. This is now possible with Scraping Fish API without compromising our commitment to keep our product usability as simple as possible.

Scraping Airbnb

Mateusz Buda
Mateusz Buda
Co-Founder
Paweł Kobojek
Paweł Kobojek
Co-Founder
Scraping Airbnb

San Francisco Bay Area, due to it's tech business importance as well as numerous iconic landmarks is a common destination for Airbnb guests. Have you ever wondered how does the hosting landscape look like there? What is the price for a stay you should expect to pay (or charge as a host)? These questions can be answered using web scraping and basic data analysis skills. We will show you how to effortlessly scrape Airbnb even though it's using JavaScript to render its content and showcase some basic data exploration.

Scraping Instagram

Mateusz Buda
Mateusz Buda
Co-Founder
Paweł Kobojek
Paweł Kobojek
Co-Founder

This blog post is a comprehensive tutorial for scraping public Instagram profile information and posts using Scraping Fish API. We will be scraping posts from a profile that lists old houses for sale to find the best deal.