JS Scenario

In this guide, we will look at how to use JS Scenario to perform activities on the scraped website.

Scraping Fish API allows you to specify a series of steps to execute once the page is loaded. You can use it, for example, to click a button or fill in a form. Steps to perform are passed as JSON in js_scenario query parameter.

Remember to encode this parameter like in the examples below.

Example

To give you an idea on how you can use this feature, let's see an example scenario which, once the page is loaded, waits for 1 second (1000 ms), clicks on the item selected by p > a CSS selector and waits for navigation to complete.

  • Name
    steps
    Type
    array
    Description

    An array of objects which define action to execute. Steps are executed in a sequence. In this example we use:

    • wait - waits for a given amount of milliseconds.
    • click_and_wait_for_navigation - clicks an element specified by the given selector and waits for navigation to complete.
  • Name
    url
    Type
    string
    Description

    URL to navigate to. This is a standard parameter required even if you don't perform any actions.

  • Name
    api_key
    Type
    string
    Description

    Your Scraping Fish API key. Required to authenticate your requests.

Execute JS Scenario

GET
/api/v1/
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "js_scenario": json.dumps({
    "steps": [
      {"click_and_wait_for_navigation": "p > a"}
    ]
  })
}

response = requests.get("https://scraping.narf.ai/api/v1/", params=payload)

Response


<!doctype html>
<html>
<head>
<title>Example Domains</title>
…
</head>
<body>
…
</body>
</html>

Steps

steps is a list of objects each of which defines an action to execute in a JS scenario. Each object's only key is a name of the action to perform and the value is its argument. For example:

{
  "steps": [
    {
      "wait_for": "#button-id"
    },
    {
      "select": {
        "selector": "#select-id",
        "options": "value1"
      }
    },
    {
      "click": "#button-id"
    }
  ]
}

Execution of this scenario will start with waiting until #button-id element is available, select value1 option from the select element (drop-down list) with #select-id id and then click the button.

In the following section, we provide all available predefined actions which you can use as steps in a JS scenario.

If you need to execute custom JavaScript code, use evaluate action.

Available actions

  • Name
    click
    Type
    string
    Description
    Clicks an element specified by a selector.
  • Name
    click_if_exists
    Type
    string
    Description
    Clicks an element specified by a selector but only if the element exists and skips this step otherwise. It can be useful if you want to close a cookie banner or other model window which does not appear every time.
  • Name
    click_and_wait_for_navigation
    Type
    string
    Description
    Clicks an element specified by a selector and waits for the navigation to complete.
  • Name
    input
    Type
    object
    Description
    Fills in given values to the input elements specified by selectors. It's an object mapping selectors to desired input values. If the order of filling in the inputs matters in your use case, you should specify each input field as a separate input action. You can optionally specify an option to "humanize" an input action. If set, actual key press events are sent. It may only be necessary if keyboard events are handled differently than usual input.
  • Name
    select
    Type
    object
    Description
    Selects option(s) from a given <select> element (drop-down list). The argument for this action must be an object with "selector" specifying the selector to find a desired <select> element and "options" (string or array) specifying the options. Selecting multiple options is supported by using an array instead of a string.
  • Name
    set_local_storage
    Type
    object
    Description
    Sets key/value pairs in localStorage. Provided object's keys to values will be resembled set in localStorage.
  • Name
    scroll
    Type
    integer
    Description
    Scrolls the web page vertically by a given number of pixels.
  • Name
    wait
    Type
    integer | object
    Description
    Waits for a fixed amount of time, specified in milliseconds. The argument for this action must be either a number or an object for random wait configuration. You may specify a range to randomize the time of wait. To do so, specify a config object with min_ms and max_ms values.
  • Name
    wait_for
    Type
    string | object
    Description
    Waits for an element specified by a selector to become visible (default) or attached. The argument for this action must be a string and a valid selector or an object with "selector" and "state" keys, where "state" is one of "visible" or "attached". If "state" is set to "visible" (default) the element you want to wait for must have non-empty bounding box (i.e. no "display: none") and no "visible: hidden". If you want to wait for an element to be present in DOM (but not necessarily visible), use "state": "attached".
  • Name
    wait_for_any
    Type
    array[string | object]
    Description
    Waits for any of the specified elements to become visible (default) or attached. If you need to wait for any of the specified elements to be visible, you can use a simpler form and only provide selectors.
  • Name
    evaluate
    Type
    string
    Description
    If the predefined actions we provide don't fit your needs and you want to evaluate custom JavaScript, this is a special action which you can use to execute arbitrary JavaScript code.

Click

{
  "steps": [
    {"click": "#a-button"}
  ]
}

Click if exists

{
  "steps": [
    {"click_if_exists": "#a-button"}
  ]
}

Click and wait for navigation

{
  "steps": [
    {"click_and_wait_for_navigation": "#a-button"}
  ]
}

Input

{
  "steps": [
    {
      "input": {
        "#input1": "value1",
        "#input2": "value2"
      }
    }
  ]
}

Select

{
  "steps": [
    {
      "select": {
        "selector": "#select1",
        "options": "1"
      }
    }
  ]
}

Set localStorage values

{
  "steps": [
    {
      "set_local_storage": {
        "key1": "value1",
        "key2": "value2"
      }
    }
  ]
}

Scroll

{
  "steps": [
    {"scroll": 1000}
  ]
}

Wait for timeout

{
  "steps": [
    {"wait": 1000}
  ]
}

Wait for selector

{
  "steps": [
    {"wait_for": "#some-button"}
  ]
}

Wait for any

{
  "steps": [
    {
      "wait_for_any": ["#some-button", "#some-other-button"]
    }
  ]
}

Custom JavaScript evaluation

{
  "steps": [
    {
      "evaluate": "console.log('Hello from Scraping Fish!')"
    }
  ]
}

Timeout

All the steps from your JavaScript scenario must complete within single trial timeout, otherwise the request will time out.