How IPs for web scraping are sourced

Real life Web Scraping requires IP addresses. They can be obtained by different means, some of which are certainly not ethical. Unfortunately, a large number of proxy providers and web scraping APIs source their IPs in unethical ways.

Below we'll show you how IPs may be sourced and why those approaches are unethical.

Why is it important?

Web Scraping is an industry by itself. People and businesses around the world make a living out of it by creating real value and contributing to the society. We find it important to keep it professional and respectful to the Internet's worldwide community. Ignoring the fact that a lot of IPs in this field are used without knowledge and consent of the people contributes to the bad image of Web Scraping. Multitude of proxy and Web Scraping API providers do not care where they get their IPs and benefit from it.

How do we know about these practices?

This is not a secret nor restricted knowledge. Part of the issue is that everyone knows about it but almost no one cares to oppose it. One of the credible sources on the topic is research of Xianghang Mi and his colleagues. We refer to their research extensively and are thankful for their contribution to the science community.

In the article Resident Evil: Understanding Residential IP Proxy as a Dark Service, the researchers checked residential proxy providers and analyzed over 6 million IP addresses. According to their study, some of the IPs were collected from compromised network or IoT devices such as routers or webcams.

These IPs are not obtained with consent of the user as providers utilizing them do not provide their software for such devices.

Based on the article Your Phone is My Proxy: Detecting and Understanding Mobile Proxy Networks, some of other mobile proxy providers use mobile SDKs to gather their IPs in an unethical way. They approach mobile apps developers (e.g. by email) and offer them a payment in exchange for integrating their SDKs. An example amount is $500 per 10,000 users per month which might be a lucrative opportunity for the owner of the app. After the integration of the SDK, the developer doesn't have control over how the proxy provider uses their users' bandwidth. In theory, users of the app agree to be part of such a proxy pool as it is stated in the Terms of Service. In practice, however, they often don't do so knowingly.

A similar story can be told about browser extensions. Browser extensions developers are emailed by proxy providers, similarly to mobile apps developers, offering money in exchange for users' bandwidth.

We find a “consent“ hidden inside the ToS without more vocal information to the user that their bandwidth is used as part of a proxy pool to be an act of treating users without respect.

It is, again, a similar approach to getting IPs preying on uninformed users. Some providers ship a free VPN service. In exchange, users of such a service, allow some of their bandwidth to be used by the provider. Compared to other VPN services they usually provide poor quality, e.g. no encryption, all requests are tracked and being logged etc.

Similarly to previous ways, in theory, people accept that they provide bandwidth to the pool. In practice, however, they are not properly informed about the fact and what it means.

Web Scraping APIs are a thin layer over the proxies. Building a Web Scraping API is much easier than building a large, international proxy pool. However, under the hood these APIs usually use the same proxy providers, effectively supporting and reinforcing bad, unethical practices. This is further reflected in their pricing schemas which are direct consequences of their underlying proxy providers of questionable ethics.