Cookies

Scraping Fish API allows to both set cookies for the response and retrieve cookies set by the website.

Set Cookies

Cookies can be passed as array in cookies parameter.

Remember to encode this parameter like in the examples below.

Cookies syntax

Cookies is an array of objects. "name" and "value" are required fields, other are optional. Example array with two cookies:

[
  {
    "name": "cookie1",
    "value": "cookie1value"
  },
  {
    "name": "cookie2",
    "value": "cookie2value",
    "domain": "example.com",
    "path": "/",
    "expires": 100000,
    "secure": true,
    "httpOnly": false,
    "sameSite": "Lax"
  }
]

Required attributes

  • Name
    name
    Type
    string
    Description

    Cookie's name.

  • Name
    value
    Type
    string
    Description

    Cookie's value.

Optional attributes

  • Name
    domain
    Type
    string
    Description

    Cookie's domain. You can either set domain and path or url. If you don't set any of the three, Scraping Fish uses domain equal to the domain of the desired URL and path to /. For example, if you scrape https://example.com/some/path?arg1=value1&arg2=value2 and don't specify cookie's URL, domain nor path, then cookie will have its domain set to example.com and path to /. See more info on MDN.

  • Name
    path
    Type
    string
    Description

    Cookie's path. You can either set domain and path or url. If you don't set any of the three, Scraping Fish uses domain equal to the domain of the desired URL and path to /. For example, if you scrape https://example.com/some/path?arg1=value1&arg2=value2 and don't specify cookie's URL, domain nor path, then cookie will have its domain set to example.com and path to /. See more info on MDN.

  • Name
    url
    Type
    string
    Description

    Cookie's URL. You can either set domain and path or url. If you don't set any of the three, Scraping Fish uses domain equal to the domain of the desired URL and path to /. For example, if you scrape https://example.com/some/path?arg1=value1&arg2=value2 and don't specify cookie's URL, domain nor path, then cookie will have its domain set to example.com and path to /. See more info on MDN.

  • Name
    expires
    Type
    integer
    Description

    Cookie's expiry timestamp. Specifies cookie lifetime in Unix time in seconds. See more info about cookie's lifetime on MDN.

  • Name
    secure
    Type
    bool
    Description

    See more info on MDN.

  • Name
    httpOnly
    Type
    bool
    Description

    See more info on MDN.

  • Name
    sameSite
    Type
    "Strict" | "Lax" | "None"
    Description

    See more info on MDN.

Example

The following example sets two cookies:

  • cookie1-name: cookie1-value,
  • cookie2-name: cookie2-value.
GET
/api/v1/
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "cookies": json.dumps(
    [
      {"name": "cookie1-name", "value": "cookie1-value"},
      {"name": "cookie2-name", "value": "cookie2-value"},
    ]
  ),
}

response = requests.get("https://scraping.narf.ai/api/v1/", params=payload)
print(response.content)

Get cookies

You can inspect Sf-Cookies header for all cookies set by the website, including dynamically set cookies. Different cookies are separated with newline character \n and cookie elements (value, domain, path, etc.) are separated with ;.

Example

GET
/api/v1/
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "cookies": json.dumps(
    [
      {"name": "cookie1-name", "value": "cookie1-value"},
      {"name": "cookie2-name", "value": "cookie2-value"},
    ]
  ),
}

response = requests.get("https://scraping.narf.ai/api/v1/", params=payload)
print(response.headers)

The output will include the following header:

Sf-Cookies: "cookie1-name=cookie1-value; expires=Wed, 31 Dec 1969 23:59:59 GMT; domain=example.com; path=/; SameSite=Lax\ncookie2-name=cookie2-value; expires=Wed, 31 Dec 1969 23:59:59 GMT; domain=example.com; path=/; SameSite=Lax"