Scraper and crawler built with Playwright and Cheerio

Go to file

Kиро.Kрика 5dc63e2952 Delete bfs-scrape.js was removed in favor of BFS.js an improved version with working precise crawling		2024-08-14 22:48:05 +03:00
BFS.js	Updated the BFS scraper to now properly work	2024-08-14 22:44:38 +03:00
LICENSE	Initial commit	2024-08-14 20:47:49 +03:00
README.md	Adding instructions	2024-08-14 21:23:25 +03:00
scrape-everything.js	added the clusterfuck	2024-08-14 20:53:42 +03:00
scrape-within-domain-only.js	added domain scope only scraper	2024-08-14 20:54:57 +03:00

README.md

Playwright_Scraper

Scraper and crawler built with Playwright and Cheerio

Versions and Differences

BFS version The BFS version uses the Breadth-First Search Approach To ensure the crawler explores all pages more thoroughly the crawler processes all immediate links (siblings) at the current depth level before moving on to deeper levels.

Scrape Everything This pretty much lets the crawler to go wild (can't recommend)

Scrape Domain Scope only Scrapes within the domain scope (worse BFS version as this goes in a straight line and doesn't scan everything)

Requirements

first install npm

Arch

sudo pacman -Sy nodejs

Debian/Ubuntu

curl -sL https://deb.nodesource.com/setup_18.x -o nodesource_setup.sh

sudo bash nodesource_setup.sh

sudo apt install nodejs

Then install Playwright and the other dependencies

npm init playwright@latest

npm install path

npm install url

npm install cheerio

npm install fs