Scraping Amazon products using scripts

A practical example of a script exploring Amazon.

Example Script:

let {newPage, end, except, extract, extractAndSave, store, runStore, waitFor} = __sandbox;
let {params, } = OPTIONS;
(async () => { try {
	// -- START --
	const page = await newPage()
	console.log('fetching urls')
	await page.goto(params.url)
	const {urls} = await page.extract('url-scraper')
	for(let url of urls ){
		console.log(`saving data of product ${url}`)
		try {
			await extractAndSave('product', url, (record) => {
				return record.asin
			})
		}
		catch{
			console.error('failed to save')
		}
	}

	// -- END --
	end()
} catch(e) { except(e) } })();

Scrapex.ai Amazon Script

Downloadable Amazon Project

Code Breakdown

The given snippet:

  • Opens a new page using the newPage() method
  • Navigate to the url provided in params as url key
  • Extract all the data marked by the scraper with url-scraper alias. This extracts all urls of products from a product list page.
  • For each Url:
    • Call the extractAndSave method that will use the scraper marked by the product alias. The id for the record being the ASIN extracted the scraper. The scraper also extracts the rating, price and name of the product.
    • On failures to save or extract, log this failure.

Scrapex.ai Amazon Store

NOTE: This script makes use of extractAndSave. This saves the generated data in a store called scraper-ScraperID