Scraping URLs containing query parameters
If the URL you want to scrape contains query parameters, it is required to first URL-encode it. This is important because otherwise our API wouldn't be able to differentiate between query params used in Scraping Fish API call vs. query params you want to pass to the desired web page.
- cURL
- Python
- JavaScript
curl --data-urlencode 'url=https://example.com?example=param&second=parameter' \
https://scraping.narf.ai/api/v1/?api_key=[your API key]
import requests
from urllib.parse import quote_plus
url = quote_plus("https://example.com?example=param&second=parameter")
api_key = "[your API key]"
print(requests.get(f"https://scraping.narf.ai/api/v1/?render_js=true&api_key={api_key}&url={url}").content)
import axios from "axios";
const url = encodeURIComponent("https://example.com?example=param&second=parameter")
const apiKey = "[your API key]"
const response = await axios.get(`https://scraping.narf.ai/api/v1/?render_js=true&api_key=${apiKey}&url=${url}`);
console.log(response.data);
info
It is a good practice to always encode the URL, regardless of using query params or not.
Below is a list of links to documentation for URL encoding methods in selected popular programming languages:
- Python: urllib.parse.quote_plus
- JavaScript: encodeURIComponent
- Java: java.net.URLEncoder.encode
- Ruby: URI.escape
- Rust: urlencoding::encode
- GO: url.QueryEscape
- PHP: urlencode
You can also use the form below for URL encoding: