GetScraping API Examples - Node.js

First, let's initialize the GetScraping client:

import { GetScrapingClient, GetScrapingParams } from 'getscraping';
import cheerio from 'cheerio';
 
const client = new GetScrapingClient('your_api_key_here');

1. Basic GET request with retry configuration

This example demonstrates a simple GET request with retry logic and a custom timeout.

import cheerio from 'cheerio';
 
const basicGetRequest = async (url: string): Promise<string> => {
  const options: GetScrapingParams = {
    url,
    method: 'GET',
    retry_config: {
      num_retries: 3,
      success_status_codes: [200, 201],
    },
    timeout_millis: 30000,
  };
  
  const response = await client.scrape(options);
  const html = await response.text();
  const $ = cheerio.load(html);
  return $('body').text();
};

2. POST request with custom headers and body

This example shows how to make a POST request with custom headers and a JSON body.

const postRequestWithHeaders = async (url: string, data: any): Promise<any> => {
  const options: GetScrapingParams = {
    url,
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Custom-Header': 'CustomValue',
    },
    body: JSON.stringify(data),
    omit_default_headers: true,
  };
  
  const response = await client.scrape(options);
  return await response.json();
};

3. Using JavaScript rendering with wait conditions

This example demonstrates how to use JavaScript rendering with a wait condition for a specific selector.

import cheerio from 'cheerio';
 
const renderJsWithWait = async (url: string): Promise<string> => {
  const options: GetScrapingParams = {
    url,
    method: 'GET',
    js_rendering_options: {
      render_js: true,
      wait_for_selector: '#dynamic-content',
      wait_millis: 5000,
    },
  };
  
  const response = await client.scrape(options);
  const html = await response.text();
  const $ = cheerio.load(html);
  return $('#dynamic-content').html() || '';
};

4. Intercepting a specific request

This example shows how to intercept a specific API request during page load.

const interceptRequest = async (url: string): Promise<any> => {
  const options: GetScrapingParams = {
    url,
    method: 'GET',
    js_rendering_options: {
      render_js: true,
      intercept_request: {
        intercepted_url_regex: '.*api\/data.*',
        intercepted_url_method: 'POST',
        request_number: 1,
        return_json: true,
      },
    },
  };
  
  const response = await client.scrape(options);
  return await response.json();
};

5. Using programmable browser actions

This example demonstrates how to use programmable browser actions to interact with the page.

import cheerio from 'cheerio';
 
const programmableBrowserActions = async (url: string): Promise<string> => {
  const options: GetScrapingParams = {
    url,
    method: 'GET',
    js_rendering_options: {
      render_js: true,
      programmable_browser: {
        actions: [
          { type: 'click', selector: '#load-more-button', wait_millis: 2000 },
          { type: 'scroll', selector: 'body', wait_millis: 1000 },
          { type: 'execute_js', javascript: 'window.scrollTo(0, document.body.scrollHeight);', wait_millis: 2000 },
        ],
      },
    },
  };
  
  const response = await client.scrape(options);
  const html = await response.text();
  const $ = cheerio.load(html);
  return $('.content').text();
};

6. Using different proxy types

This example shows how to use different proxy types for your requests.

import cheerio from 'cheerio';
 
const proxyRequest = async (url: string): Promise<string> => {
  const options: GetScrapingParams = {
    url,
    method: 'GET',
    use_residential_proxy: true,
    // Alternatively:
    // use_isp_proxy: true,
    // use_mobile_proxy: true,
    // use_own_proxy: 'http://username:password@proxy.example.com:8080',
  };
  
  const response = await client.scrape(options);
  const html = await response.text();
  const $ = cheerio.load(html);
  return $('body').html() || '';
};

7. Handling cookies

This example demonstrates how to handle cookies in your requests and responses.

import cheerio from 'cheerio';
 
const cookieHandling = async (url: string, initialCookies: string[]): Promise<{ content: string; cookies: string[] }> => {
  const options: GetScrapingParams = {
    url,
    method: 'GET',
    cookies: initialCookies,
  };
  
  const response = await client.scrape(options);
  const html = await response.text();
  const $ = cheerio.load(html);
  const newCookies = response.headers.raw()['set-cookie'];
  return { content: $('body').text(), cookies: newCookies };
};

8. Using custom success criteria

This example shows how to use custom success criteria for your requests.

import cheerio from 'cheerio';
 
const customSuccessCriteria = async (url: string): Promise<string> => {
  const options: GetScrapingParams = {
    url,
    method: 'GET',
    retry_config: {
      num_retries: 3,
      success_status_codes: [200],
      success_selector: '#content-loaded',
    },
  };
  
  const response = await client.scrape(options);
  const html = await response.text();
  const $ = cheerio.load(html);
  return $('#content-loaded').text();
};

9. Combining multiple features

This example demonstrates how to combine multiple features of the GetScraping API in a single request.

const complexRequest = async (url: string): Promise<any> => {
  const options: GetScrapingParams = {
    url,
    method: 'GET',
    js_rendering_options: {
      render_js: true,
      wait_for_selector: '#app-loaded',
      programmable_browser: {
        actions: [
          { type: 'click', selector: '#accept-cookies', wait_millis: 1000 },
          { type: 'execute_js', javascript: 'document.querySelector("#load-more").click();', wait_millis: 2000 },
        ],
      },
    },
    use_residential_proxy: true,
    retry_config: {
      num_retries: 5,
      success_status_codes: [200],
      success_selector: '#all-content-loaded',
    },
    timeout_millis: 60000,
    response_type: 'json',
  };
  
  const response = await client.scrape(options);
  return await response.json();
};

These examples showcase various features of the GetScraping API, including basic requests, JavaScript rendering, request interception, programmable browser actions, proxy usage, cookie handling, custom success criteria, and combining multiple features in a single request. They also demonstrate the use of Cheerio for HTML parsing when dealing with HTML responses.

Remember to install the necessary packages if you haven't already:

npm install get-scraping cheerio

yarn add get-scraping cheerio

Request Options Python