Skip to main content

Scraping URLs containing query parameters

If the URL you want to scrape contains query parameters it is required to first url encode it. This is important, because otherwise our API wouldn't be able to differentiate between query params used in Scraping Fish API call vs. query params you want to pass to the desired web page.

import requests
from urllib.parse import quote_plus

url = quote_plus("https://example.com?example=param&second=parameter")
api_key = "[your api key]"

print(requests.get(f"https://scraping.narf.ai/api/v1/?render_js=true&api_key={api_key}&url={url}").content)

It is a good practice to always encode the URL, regardless of using query params or not.