Skip to main content

Cookies

Set cookies​

Cookies can be passed as array in cookies parameter.

tip

Remember to encode this parameter like in the examples below.

Cookies syntax​

Cookies is an array of objects. "name" and "value" are required fields, other are optional. Example array with two cookies:

[
  {
    "name": "cookie1",
    "value": "cookie1value"
  },
  {
    "name": "cookie2",
    "value": "cookie2value",
    "domain": "example.com",
    "path": "/",
    "expires": 100000,
    "secure": true,
    "httpOnly": false,
    "sameSite": "Lax"
  }
]

name​

Required string. Cookie's name.

value​

Required string. Cookie's value.

domain​

Optional string. Cookie's domain.

You can either set domain and path or url. If you don't set any of the three, we use domain equal to the domain of the desired URL and path to /. For example, if you scrape https://example.com/some/path?arg1=value1&arg2=value2 and don't specify cookie's URL, domain nor path, then cookie will have its domain set to example.com and path to /. See more info on MDN.

path​

Optional string. Cookie's path.

You can either set domain and path or url. If you don't set any of the three, we use domain equal to the domain of the desired URL and path to /. For example, if you scrape https://example.com/some/path?arg1=value1&arg2=value2 and don't specify cookie's URL, domain nor path, then cookie will have its domain set to example.com and path to /. See more info on MDN.

url​

Optional string. Cookie's URL.

You can either set domain and path or url. If you don't set any of the three, we use domain equal to the domain of the desired URL and path to /. For example, if you scrape https://example.com/some/path?arg1=value1&arg2=value2 and don't specify cookie's URL, domain nor path, then cookie will have its domain set to example.com and path to /. See more info on MDN.

expires​

Optional integer. Cookie's expiry timestamp.

Specifies cookie lifetime in Unix time in seconds. See more info about cookie's lifetime on MDN.

secure​

Optional boolean.

See more info on MDN.

httpOnly​

Optional boolean.

See more info on MDN.

sameSite​

Optional string. One of "Strict", "Lax", or "None".

Example​

The following example sets two cookies:

  • cookie1-name: cookie1-value,
  • cookie2-name: cookie2-value.
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "cookies": json.dumps(
    [
      {"name": "cookie1-name", "value": "cookie1-value"},
      {"name": "cookie2-name", "value": "cookie2-value"},
    ]
  ),
}

response = requests.get("https://scraping.narf.ai/api/v1/", params=payload)
print(response.content)

Get cookies​

You can inspect Sf-Cookies header for all cookies set by the website, including dynamically set cookies. Different cookies are separated with newline character \n and cookie elements (value, domain, path, etc.) are separated with ; .

Example​

This example sets custom cookies which will be avialble in the Sf-Cookies header.

import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "cookies": json.dumps(
    [
      {"name": "cookie1-name", "value": "cookie1-value"},
      {"name": "cookie2-name", "value": "cookie2-value"},
    ]
  ),
}

response = requests.get("https://scraping.narf.ai/api/v1/", params=payload)
print(response.headers)

The output will include the following header:

Sf-Cookies: "cookie1-name=cookie1-value; expires=Wed, 31 Dec 1969 23:59:59 GMT; domain=example.com; path=/; SameSite=Lax\ncookie2-name=cookie2-value; expires=Wed, 31 Dec 1969 23:59:59 GMT; domain=example.com; path=/; SameSite=Lax"