Scraping Trip Advisor hotels using Scripts

A practical example of a script exploring Trip Advisor.

Example Script:

let {newPage, end, except, extract, extractAndSave, store, runStore, waitFor} = __sandbox;
let {params, } = OPTIONS;
(async () => { try {
	// -- START --
	let id = 0
	const extractData = async (page) => {
		console.log('extracting data')
		const {hotel} = await page.extract('ta-scraper')
		return hotel.map((hotelData) => {
			const {price, hotel:name} = hotelData
			return({
				id: id++,
				name,
				price
			})
		})
	}

	const page = await newPage()
	await page.goto(params.url)
	console.log(`nav to ${params.url}`)
	let data = await extractData(page)
	console.log('navigating to next page')
	await page.clickTag('ta-scraper', 'next_btn')
	await waitFor(2000); 
	data.push(...await extractData(page))
	console.log('saving data')
	await store.saveMany('ta-store',data)

	// -- END --
	end()
} catch(e) { except(e) } })();

Scrapex.ai Trip Advisor Script

Downloadable TripAdvisor Project

Code Breakdown

The given snippet:

  • Opens a new page using the newPage() method
  • Navigate to the url provided in params as url key
  • Call Extract Data function. This function:
    • Extracts from page the data given by the scraper marked by the alias ta-scraper. This scraper in this context extracts the price and names of hotels from Trip Advisor.
    • This then generates a store Blob to save the extracted data.
  • Click next page button for pagination
  • Wait 2s for navigation to finish
  • Expand generated object list
  • Store all data to project-level store with the ta-store

NOTE: This script makes use of scraper references.