Crawlee's Proxy Rotation and Request Queue: How a 25K-Star Scraper Library Handles Agent-Scale Data Extraction

Crawlee is a Node.js scraping library with 25,345 stars that treats web extraction as a queueing and concurrency problem. It ships with request deduplication, proxy rotation, session management, and pluggable storage adapters. Agents that need structured web data at scale need more than a headless browser wrapper. They need retry logic, fingerprint evasion, and crash-safe state.

Crawlee was built by Apify, a commercial scraping platform, and open-sourced as infrastructure for AI data pipelines, RAG ingestion, and LLM training sets. It supports Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. The library handles the plumbing so your agent can focus on extraction logic.

Request Queue Architecture

Crawlee’s RequestQueue is a persistent, deduplicated FIFO queue with retry semantics. Every request gets a unique ID derived from its URL and method. The queue tracks three states: pending, in-progress, and handled.

Key behaviors:

• Deduplication: Adding the same URL twice is a no-op. The queue hashes url + method + payload to generate a fingerprint.
• Crash recovery: Requests marked in-progress are reclaimed after a configurable timeout (default 60 seconds). If your scraper dies mid-request, the queue assumes the worker failed and re-enqueues.
• Priority: You can assign numeric priorities. Higher values jump the queue.
• Persistence: The default storage adapter writes to disk in ./storage/request_queues/. You can swap in a remote adapter (Apify Cloud, Redis, S3) for distributed crawls.

Retry flow:

1. Request is popped from the queue and marked in-progress.
2. Handler executes. If it throws, Crawlee increments retryCount.
3. If retryCount < maxRequestRetries (default 3), the request goes back to pending with exponential backoff.
4. If retries are exhausted, the request is marked failed and logged.

This is not a generic job queue. It is purpose-built for URL crawling. You cannot enqueue arbitrary JSON payloads or task types. Every item is a Request object with a URL, headers, and optional user data.

Autoscaled Pool vs. Simple Worker Queue

Crawlee’s AutoscaledPool dynamically adjusts concurrency based on system load. It monitors CPU, memory, and event loop lag. If your scraper starts thrashing, the pool reduces parallelism. If resources are idle, it spins up more workers.

Comparison:

Feature	AutoscaledPool	Simple Worker Queue
Concurrency control	Dynamic (1 to maxConcurrency)	Fixed
Backpressure detection	CPU, memory, event loop lag	None
Crash recovery	Reclaims timed-out tasks	Manual implementation
Rate limiting	Built-in per-domain throttling	External library required
Agent workload fit	High (adapts to bursty extraction)	Low (fixed parallelism wastes resources or overloads)

For agent workloads, autoscaling matters. An agent extracting product data from 10,000 e-commerce pages does not know in advance how many pages will trigger JavaScript rendering (slow) versus static HTML (fast). AutoscaledPool adjusts on the fly.

Configuration snippet:

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
  maxConcurrency: 50,
  minConcurrency: 1,
  autoscaledPoolOptions: {
    desiredConcurrency: 10,
    maxConcurrencyPerCpu: 5,
    systemStatusOptions: {
      maxUsedMemoryRatio: 0.85,
      maxUsedCpuRatio: 0.90,
    },
  },
  async requestHandler({ request, page, enqueueLinks }) {
    const title = await page.title();
    await enqueueLinks({ globs: ['https://example.com/products/*'] });
    console.log(`Scraped: ${title}`);
  },
});

await crawler.run(['https://example.com']);

The pool starts at 10 workers. If CPU usage stays below 90% and memory below 85%, it scales up to 50. If either threshold is breached, it scales down.

Proxy Rotation and Session Management

Crawlee’s proxy rotation is session-aware. A session is a logical grouping of requests that share the same IP, cookies, and browser fingerprint. This prevents sites from detecting bot behavior when you switch IPs mid-session.

Proxy strategies:

• No rotation: All requests use the same proxy or direct connection.
• Per-request rotation: Every request gets a new IP. Fast but breaks session continuity.
• Session-based rotation: Requests within a session reuse the same IP. Sessions rotate after a configurable number of requests or time window.

Session pool behavior:

• Crawlee maintains a pool of sessions. Each session has a usageCount and errorScore.
• If a session hits a CAPTCHA or 403, its error score increases. After a threshold, the session is retired and a new one is created.
• Sessions persist cookies and local storage. If you log into a site, subsequent requests in that session stay authenticated.

Proxy configuration:

import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
  proxyUrls: [
    'http://proxy1.example.com:8000',
    'http://proxy2.example.com:8000',
  ],
  sessionPoolOptions: {
    maxPoolSize: 100,
    sessionOptions: {
      maxUsageCount: 50,
      maxErrorScore: 3,
    },
  },
});

const crawler = new PlaywrightCrawler({
  proxyConfiguration,
  useSessionPool: true,
  async requestHandler({ request, page, session }) {
    console.log(`Using session: ${session.id}`);
    const html = await page.content();
  },
});

Each session makes up to 50 requests before rotating. If three requests fail, the session is discarded.

Agent implications:

• Agents scraping authenticated content (e.g., LinkedIn profiles, Salesforce dashboards) need session persistence. Crawlee’s session pool handles this without manual cookie management.
• Agents hitting rate-limited APIs benefit from per-session throttling. Crawlee can enforce maxRequestsPerMinute per session, not globally.

Storage Adapters and Agent Memory Integration

Crawlee’s storage layer is pluggable. The default adapter writes to disk. You can swap in adapters for Apify Cloud, AWS S3, or custom backends.

Storage types:

• Request Queue: Stores pending, in-progress, and handled requests.
• Dataset: Append-only storage for scraped items. Each item is a JSON object.
• Key-Value Store: Generic key-value pairs. Used for snapshots, screenshots, or intermediate state.

Dataset example:

import { PlaywrightCrawler, Dataset } from 'crawlee';

const crawler = new PlaywrightCrawler({
  async requestHandler({ request, page }) {
    const title = await page.title();
    const url = request.url;
    
    await Dataset.pushData({ title, url, scrapedAt: new Date() });
  },
});

await crawler.run(['https://example.com']);

const dataset = await Dataset.open();
const items = await dataset.getData();
console.log(items.items); // Array of scraped objects

Agent memory integration:

• Vector stores: Export dataset items to Pinecone, Weaviate, or Qdrant. Crawlee does not have native vector store adapters, but you can write a custom storage client that pushes to your vector DB after each scrape.
• Graph databases: Store crawled links and relationships in Neo4j. Use the key-value store to checkpoint graph state.
• LLM context windows: Crawlee’s dataset format is JSON. Pipe it directly into LangChain’s document loaders or LlamaIndex ingestion pipelines.

Custom storage adapter:

import { StorageClient } from '@crawlee/core';

class VectorStoreAdapter extends StorageClient {
  async pushData(data) {
    // Push to Pinecone, Weaviate, etc.
    await this.vectorStore.upsert(data);
  }
}

const crawler = new PlaywrightCrawler({
  storageClient: new VectorStoreAdapter(),
  // ...
});

Crawlee does not ship with vector store adapters out of the box. You write the glue code.

Failure Modes and Observability

Common failure modes:

1. Proxy ban: All proxies in the pool get blocked. Crawlee retries until maxRequestRetries is exhausted, then fails the request. You need external proxy monitoring.
2. Memory leak in Playwright: Long-running crawls with Playwright can leak browser contexts. Crawlee’s autoscaler detects high memory usage and reduces concurrency, but it does not restart the browser. You need to set maxRequestsPerCrawl or restart the crawler periodically.
3. Session pool exhaustion: If all sessions hit CAPTCHAs, the pool stalls. Crawlee does not have built-in CAPTCHA solving. You need to integrate a service like 2Captcha or Anti-Captcha.
4. Request queue corruption: If the storage adapter crashes mid-write, the queue can enter an inconsistent state. The default disk adapter is not transactional. Use a remote adapter with ACID guarantees for production.

Observability hooks:

Crawlee emits events for request success, failure, and retry. You can attach listeners to push metrics to Prometheus, Datadog, or CloudWatch.

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
  async requestHandler({ request, page }) {
    // Scraping logic
  },
  async failedRequestHandler({ request, error }) {
    console.error(`Failed: ${request.url}`, error);
    // Push to error tracking service
  },
});

crawler.on('requestSucceeded', ({ request }) => {
  console.log(`Success: ${request.url}`);
});

crawler.on('requestFailed', ({ request, error }) => {
  console.log(`Failed: ${request.url}`, error.message);
});

Crawlee does not have built-in tracing or distributed logging. You need to instrument handlers manually.

Crawlee vs. Point-and-Click Browser Automation

Dimension	Crawlee	Point-and-Click Tools (e.g., Puppeteer Raw)
Request queue	Built-in, persistent, deduplicated	Manual implementation
Proxy rotation	Session-aware, automatic	Manual proxy cycling
Retry logic	Exponential backoff, configurable	Manual try-catch loops
Autoscaling	CPU/memory-aware concurrency	Fixed parallelism
Storage	Pluggable adapters (disk, cloud)	Manual file I/O
Agent fit	High (designed for data pipelines)	Low (designed for one-off scripts)

Crawlee is not a browser automation library. It is a scraping orchestrator that happens to support browser automation. If your agent needs to click buttons and fill forms, use Playwright directly. If your agent needs to extract structured data from 100,000 pages, use Crawlee.

Technical Verdict

Use Crawlee when:

• Your agent needs to scrape more than 1,000 pages and you want crash recovery, deduplication, and retry logic out of the box.
• You need session-aware proxy rotation to avoid IP bans or maintain authentication state.
• You want autoscaling that adapts to system load without manual tuning.
• You need pluggable storage adapters to integrate with cloud infrastructure or vector databases.

Avoid Crawlee when:

• Your agent scrapes fewer than 100 pages and you can tolerate manual retry logic.
• You need real-time browser interaction (form filling, button clicking) without data extraction. Use Playwright or Puppeteer directly.
• You need built-in CAPTCHA solving or fingerprint randomization. Crawlee does not ship with these. You need external services.
• You need transactional guarantees on the request queue. The default disk adapter is not ACID-compliant. Use a remote adapter with a database backend.

Crawlee is infrastructure for agents that treat web scraping as a batch data pipeline, not a one-off script. It handles the plumbing so your agent can focus on extraction logic and downstream processing.

Source Links

• Crawlee GitHub Repository
• Crawlee Documentation