Skip to main content

Scraping URLs containing query parameters

If the URL you want to scrape contains query parameters, it is required to first URL-encode it. This is important because otherwise our API wouldn't be able to differentiate between query params used in Scraping Fish API call vs. query params you want to pass to the desired web page.

curl --data-urlencode 'url=https://example.com?example=param&second=parameter' \
https://scraping.narf.ai/api/v1/?api_key=[your API key]
info

It is a good practice to always encode the URL, regardless of using query params or not.

Below is a list of links to documentation for URL encoding methods in selected popular programming languages:

You can also use the form below for URL encoding: