Forward HTTP headers

You can add your custom headers to a request or force original browser headers to be removed. To remove a header, set its value to empty string.

Passing headers to be forwarded

Headers are passed as a JSON in headers parameter:

Example header parameter value

{"x-custom-header": "value", "host": ""}

Example

In the example below, we add two headers:

  • Date: Wed, 24 Jan 2024 07:28:00 GMT
  • X-Custom-Header: value These headers will be forwarded to the requested page.
GET
/api/v1/
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://httpbin.org/headers",
  "headers": json.dumps(
    {"date": "Wed, 24 Jan 2024 07:28:00 GMT", "x-custom-header": "value"}
  ),
}

response = requests.get("https://scraping.narf.ai/api/v1/", params=payload)
print(response.content)

In the response from https://httpbin.org, you will be able to validate that the desired headers were indeed included in the request:

Response

{
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Accept-Language": "en-US",
    "Date": "Wed, 24 Jan 2024 07:28:00 GMT",
    "Host": "httpbin.org", 
    "Sec-Ch-Ua": "\"Google Chrome\";v=\"107\", \"Chromium\";v=\"107\", \";Not A Brand\";v=\"99\"", 
    "Sec-Ch-Ua-Mobile": "?0", 
    "Sec-Ch-Ua-Platform": "\"Windows\"", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "none", 
    "Sec-Fetch-User": "?1", 
    "Upgrade-Insecure-Requests": "1", 
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-63963404-57e4efe21185436d73ea000e", 
    "X-Custom-Header": "value"
  }
}