Skip to main content

JavaScript scenario

Scraping Fish API allows you to specify a series of steps to execute once the page is loaded. You can use it for example to click a button or fill out a form.

Steps are passed as a JSON in js_scenario parameter. Remember to encode this parameter like in the example below. Here the page will load, sleep 1 second (1000 ms), click on the item selected by p > a CSS selector and wait for navigation to be completed:

curl -G \
--data-urlencode 'url=https://example.com' \
--data-urlencode 'js_scenario={"steps": [{"wait": 1000}, {"click_and_wait_for_navigation": "p > a"}]}' \
'https://scraping.narf.ai/api/v1/?api_key=[your api key]'
info

If the action causes navigation to another url it will be charged as a separate request.

Steps

"steps" is a list of dictionaries. Each dictionary's only key is a name of the action to perform and value is its argument. For example:

{
"steps": [
{"wait_for": "#button-id"},
{"select": {"selector": "#select-id", "options": "value1"}}
{"click": "#button-id"},
]
}

Execution of this scenario will start with waiting until "#button-id" element is available, select "value1" option from the select element with #select-id id and then click the button.

Available actions

Click

Clicks an element specified by a selector. Example:

{
"steps": [
{"click": "#a-button"}
]
}

The argument must be a string and a valid selector.

caution

If clicking an element navigates to another page (e.g. by submitting a form), use "click_and_wait_for_navigation" which waits until the navigation is completed.

Click and wait for navigation

Clicks an element specified by a selector and waits for the navigation to complete. Example:

{
"steps": [
{"click_and_wait_for_navigation": "#a-button"}
]
}

The argument must be a string and a valid selector.

caution

If clicking the specified element doesn't trigger navigation, use plain "click" as using this method will hang.

Input

Types given values to the inputs specified by selectors. Example:

{
"steps": [{
"input": {
"#input1": "value1",
"#input2": "value2"
}
}]
}

The argument must be a dictionary mapping from inputs selectors to values.

note

If the order of typing inputs matters in your use case, you should specify each input field as a separate input action. For example, if #input2 should be filled before #input1:

{
"steps": [{
"input": {
"#input2": "value2"
},
"input": {
"#input1": "value1"
},
}]
}

Select

Selects option(s) from a given <select> element. Example:

{
"steps": [{
"select": {
"selector": "#select1",
"options": "1"
},
}]
}

The argument must be a dictionary with "selector" specifying the selector to find a desired <select> and "options" (string or array) specifying the options. Multiple options selection is supported by using array instead of string:

{
"steps": [{
"select": {
"selector": "#select1",
"options": ["1", "2"]
},
}]
}

Scroll

Scrolls vertically by a given number of pixels. Example:

{
"steps": [
{"scroll": 1000}
]
}

The argument must be a number.

Wait for timeout

Waits for a fixed amount of time, specified in milliseconds.

{
"steps": [
{"wait": 1000}
]
}

The argument must be a number.

Wait for an element

Waits for an element specified by a selector to become visible (default) or attached. Example:

{
"steps": [
{"wait_for": "#some-button"}
]
}

The argument must be a string and a valid selector or an object with "selector" and "state" keys, where "state" is one of "visible" or "attached". If "state" is set to "visible" (default) the element you want to wait for must have non-empty bounding box (i.e. no "display: none") and no "visible: hidden". If you want to wait for an element to be present in DOM (but not necessarily visible), use "state": "attached", for example:

{
"steps": [
{"wait_for": {"selector": "#some-button", "state": "attached"}}
]
}

Custom JavaScript evaluation

If the actions we provide don't fit your needs and you want to evaluate a custom JavaScript, there's a special evaluate action to which you can pass arbitrary JavaScript code:

{
"steps": [
{"evaluate": "console.log('Hello from Scraping Fish!')"}
]
}