JavaScript Web Crawling Not Working as Expected?
If your JavaScript crawler misses dynamic content, struggles with pagination, or gets blocked by websites, the right crawling setup can help you collect data reliably without slowing down your application.
- JavaScript crawler development
- Dynamic content extraction
- Puppeteer & Playwright setup
- Crawling performance optimization
Website crawling is an important technique for collecting information, analyzing website structures, monitoring content, and automating data extraction. Developers frequently use web crawlers to discover internal links, extract page information, and understand how websites organize their content.
If you’re wondering how to crawl a website using JavaScript, Node.js provides several tools that simplify the process. Libraries such as Axios, Cheerio, Crawlee, and Playwright allow developers to build crawlers for both static and JavaScript-rendered websites.
However, building a reliable crawler requires more than sending HTTP requests. Developers must also manage URL discovery, duplicate pages, robots.txt rules, rate limits, and dynamically generated content.
This guide explains how website crawling works, how to create a JavaScript web crawler, which tools to use, and how to avoid common crawling mistakes.
What is Website Crawling in JavaScript?
Web crawling involves visiting web pages, finding hyperlinks within them, and extracting information from them using an automated system.
In a JavaScript web crawler, the crawler starts with a URL, retrieves its HTML content, finds internal hyperlinks, and visits more pages based on those links.
This goes on until the set limits of the crawler have been reached.
JavaScript programmers often use Node.js to build crawlers because it supports asynchronous HTTP requests and HTML content retrieval.
Web crawling has applications in SEO audits, content monitoring, web migration analyses, and data extraction.
How Does a JavaScript Web Crawler Work?
A JavaScript crawler follows a structured process to discover and analyze web pages.
It usually starts with a seed URL, which serves as the crawling starting point.
The crawler retrieves the page, extracts links, adds eligible URLs to a queue, and repeats the process.
A typical crawling workflow includes:
- Start with a URL: Define the website’s starting page.
- Fetch the page: Retrieve HTML using an HTTP client or browser.
- Parse the HTML: Extract titles, links, headings, or other required data.
- Discover internal links: Identify URLs belonging to the target website.
- Avoid duplicates: Track previously visited pages.
- Continue crawling: Visit eligible URLs until the crawl limit is reached.
Developers can customize these steps depending on the website’s structure and data collection requirements.
What Tools are Needed to Crawl a Website Using JavaScript?
JavaScript offers several libraries for developing web crawlers. The appropriate tools depend on whether the website uses static HTML or dynamically rendered content.
Node.js
Node.js lets JavaScript run outside the browser.
It provides the runtime environment needed to create command-line crawlers, manage HTTP requests, and process downloaded content.
Axios
Axios is an HTTP client that helps developers request web pages and retrieve their responses.
It supports asynchronous operations, request headers, timeouts, and error handling.
Cheerio
Cheerio is a lightweight HTML parsing library.
It provides a jQuery-like interface for selecting elements, extracting links, and reading page content.
Cheerio parses HTML but does not execute JavaScript or render web pages.
Playwright
Playwright automates real browsers, making it useful for websites that generate content through JavaScript.
It can navigate pages, wait for elements, and interact with supported browser interfaces.
Crawlee
Crawlee is a crawling and scraping library for JavaScript and TypeScript.
It provides features such as request queues, URL management, concurrency controls, and integrations with browser automation tools.
For larger crawling projects, Crawlee can reduce the custom infrastructure developers need to build.
How to Crawl a Website Using JavaScript Step by Step
You can build a basic JavaScript website crawler using Node.js, Axios, and Cheerio.
The following example demonstrates how to discover internal links, extract page titles, and prevent duplicate visits.
Step 1: Install Node.js
Install a current supported version of Node.js from theofficial Node.js website.
Verify the installation:
node --version
npm --version
These commands display the installed Node.js and npm versions.
Step 2: Create a JavaScript Project
Create a directory for your crawler:
mkdir javascript-web-crawler
cd javascript-web-crawler
Initialize the project:
npm init -y
This creates a package.json file for managing dependencies and project configuration.
Step 3: Install Axios and Cheerio
Install the required libraries:
npm install axios cheerio
Axios retrieves web pages, while Cheerio parses their HTML.
Step 4: Create the Crawler Script
Create a file named crawler.js.
Add the following code:
const axios = require("axios");
const cheerio = require("cheerio");
const startUrl = "https://example.com/";
const origin = new URL(startUrl).origin;
const visited = new Set();
const queue = [startUrl];
const MAX_PAGES = 10;
async function crawl() {
while (queue.length > 0 && visited.size < MAX_PAGES) { const url = queue.shift(); if (visited.has(url)) continue; visited.add(url); try { const response = await axios.get(url, { timeout: 10000, headers: { "User-Agent": "ExampleSiteCrawler/1.0" } }); const contentType = response.headers["content-type"] || ""; if (!contentType.includes("text/html")) continue; const $ = cheerio.load(response.data); const title = $("title").first().text().trim(); console.log(`URL: ${url}`); console.log(`Title: ${title}`); $("a[href]").each((_, element) => {
const href = $(element).attr("href");
try {
const nextUrl = new URL(href, url);
nextUrl.hash = "";
if (
nextUrl.origin === origin &&
/^https?:$/.test(nextUrl.protocol) &&
!visited.has(nextUrl.href) &&
!queue.includes(nextUrl.href)
) {
queue.push(nextUrl.href);
}
} catch {
// Ignore invalid links.
}
});
} catch (error) {
console.error(`Failed: ${url} - ${error.message}`);
}
await new Promise(resolve => setTimeout(resolve, 1000));
}
}
crawl();
This educational example discovers same-origin links and visits up to ten pages.
It does not yet enforce robots.txt rules, so use it only on a website you own or are explicitly authorized to crawl, where the selected URLs are permitted.
Step 5: Run the JavaScript Crawler
Execute the script:
node crawler.js
The crawler prints the URLs and page titles it retrieves.
Example output:
URL: https://example.com/
Title: Example Domain
The actual output depends on the target website’s available links and content.
How to Extract Website Data Using Cheerio?
Beyond discovering links, developers can use Cheerio to extract specific HTML elements.
For example, the following script retrieves the page title, meta description, and H1 headings:
const axios = require("axios");
const cheerio = require("cheerio");
async function extractPageData(url) {
const response = await axios.get(url);
const $ = cheerio.load(response.data);
return {
url,
title: $("title").text().trim(),
description:
$('meta[name="description"]').attr("content") || "",
headings: $("h1")
.map((_, element) => $(element).text().trim())
.get()
};
}
extractPageData("https://example.com/")
.then(console.log)
.catch(console.error);
This approach is useful for checking SEO metadata, identifying missing headings, and analyzing page structures.
For production crawlers, add request limits, response validation, URL restrictions, and appropriate error handling.
How to Crawl JavaScript-rendered Websites Using Playwright?
Some websites load content dynamically through JavaScript frameworks such as React, Angular, or Vue.
Axios retrieves the initial HTML response but does not execute client-side JavaScript.
If important content appears only after browser execution, developers may need browser automation.
Install Playwright
Run:
npm install playwright
npx playwright install chromium
Create a Browser-based Crawler
const { chromium } = require("playwright");
async function crawlDynamicPage(url) {
const browser = await chromium.launch({
headless: true
});
try {
const page = await browser.newPage();
await page.goto(url, {
waitUntil: "domcontentloaded"
});
await page.locator("h1").first().waitFor({
timeout: 10000
}).catch(() => {});
const title = await page.title();
const links = await page.locator("a[href]").evaluateAll(
elements => elements.map(link => link.href)
);
console.log({ title, links });
} finally {
await browser.close();
}
}
crawlDynamicPage("https://example.com/");
This script launches Chromium, opens a page, and extracts available links after the page has loaded.
For dynamic websites, waiting for a specific content element is often more reliable than waiting for all network activity to stop.
Browser automation requires additional CPU and memory resources, so it should be used when ordinary HTTP requests cannot retrieve the required information.
How to Crawl Multiple Pages Using Crawlee?
For larger websites, manually managing URL queues, concurrency, and retries can become complicated.
Crawlee provides structured crawling capabilities that simplify these tasks.
Install Crawlee:
npm install crawlee
Create an ES module named crawler.mjs:
import { CheerioCrawler } from "crawlee";
const crawler = new CheerioCrawler({
maxRequestsPerCrawl: 20,
maxConcurrency: 2,
respectRobotsTxtFile: true,
async requestHandler({ request, $, enqueueLinks }) {
const title = $("title").text().trim();
console.log({
url: request.loadedUrl || request.url,
title
});
await enqueueLinks({
strategy: "same-origin"
});
}
});
await crawler.run(["https://example.com/"]);
This example uses Crawlee’s request handling and link discovery features.
The respectRobotsTxtFile option instructs Crawlee to consider robots.txt restrictions.
For production use, configure appropriate delays, retry limits, data storage, and request identification.
How to Handle Robots.txt and Crawling Restrictions?
Before crawling a website, developers should review its robots.txt file and applicable access policies.
The robots.txt file provides instructions indicating which paths automated crawlers may access.
For example:
User-agent: *
Disallow: /admin/
Disallow: /private/
A crawler matching these rules should avoid the restricted paths.
You can usually inspect a website’s robots.txt file at:
https://example.com/robots.txt
Robots.txt is not an authorization mechanism, and permission to access data must still be considered separately.
Developers should also avoid bypassing authentication, CAPTCHA systems, or technical restrictions.
For reliable and responsible crawling, use a descriptive user agent, respect access restrictions, and keep request rates reasonable.
Best Practices for Building a JavaScript Web Crawler
A reliable web crawler should collect useful information without creating unnecessary load or violating website access restrictions.
Developers should define a clear crawling objective before choosing tools and implementing extraction logic.
Important practices include:
- Respect robots.txt and access policies: Avoid prohibited or unauthorized pages.
- Limit requests: Configure crawl depth, concurrency, delays, and maximum page counts.
- Normalize URLs: Reduce unnecessary duplicate crawling.
- Handle errors: Manage timeouts, redirects, invalid HTML, and failed requests.
- Use browser automation selectively: Reserve Playwright for pages requiring JavaScript execution.
- Store structured results: Save extracted data in JSON, CSV, or databases for analysis.
For recurring crawling tasks, monitoring failures and maintaining logs can help identify changes in website behavior.
How Moon Technolabs Helps With JavaScript Development and Web Automation
Moon Technolabs provides JavaScript development and custom software development services to help businesses build scalable web applications, automation tools, and data-driven solutions. Our developers can assist with Node.js applications, API integrations, website data processing, and backend automation workflows tailored to business requirements.
Whether you need to build a custom website crawler, automate data collection, or integrate extracted information into an existing platform, Moon Technolabs can help plan and implement the right solution. Our team focuses on maintainable architecture, efficient processing, and reliable application performance.
Struggling to Crawl Dynamic Websites With JavaScript?
Our JavaScript experts help build custom web crawlers, automate data extraction, and handle dynamic content with scalable Node.js solutions.
Conclusion
Learning how to crawl a website using JavaScript allows developers to automate URL discovery, collect structured information, and analyze website content efficiently.
Node.js provides a flexible foundation for developing web crawlers, while Axios and Cheerio are useful for retrieving and parsing static HTML.
For websites that require JavaScript rendering, Playwright provides browser automation capabilities. Crawlee offers additional features for managing larger crawling workflows.
The right approach depends on the website’s architecture, the information being collected, and the scale of the crawling project.
By respecting website restrictions, controlling request rates, handling duplicate URLs, and validating collected data, developers can build reliable JavaScript crawlers for SEO analysis, content monitoring, and authorized data collection.
Get in Touch With Us
Submitting the form below will ensure a prompt response from us.










