Skip to main content

​
What is Crawl4AI?

Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. By integrating Browser Cash’s stealth browsers with Crawl4AI, you can easily crawl challenging websites without getting blocked.

​
Setting up the integration

1. Create environment with uv (Python>=3.11): 
uv init
2. Install Crawl4AI and requests: 
uv add crawl4ai requests
uv sync
3. Get a Browser Cash API key from the dashboard and add it to your .env file: 
# .env
BROWSER_CASH_API_KEY=your-key
4. Run your Crawl4AI task, powered by Browser Cash: 
# main.py
import os
import asyncio
import requests
from crawl4ai import *

async def main():
    # 1) Create a Browser Cash session and get its CDP URL
    resp = requests.post(
        "https://api.browser.cash/v1/browser/session",
        headers={
            "Authorization": f"Bearer {os.getenv('BROWSER_CASH_API_KEY')}",
            "content-type": "application/json",
        },
        json={},
    )
    resp.raise_for_status()
    cdp_url = resp.json()["cdpUrl"]

    # 2) Configure Crawl4AI to use the Browser Cash browser
    browser_config = BrowserConfig(
        cdp_url=cdp_url,
        use_managed_browser=True,  # Enable managed browser mode
    )

    # 3) Run the crawler with the Browser Cash browser
    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://www.nbcnews.com/business",
        )
        print(result.markdown)

        # 4) Stop the Browser Cash session
        requests.delete(
            "https://api.browser.cash/v1/browser/session",
            headers={
                "Authorization": f"Bearer {os.getenv('BROWSER_CASH_API_KEY')}",
            },
            params={"sessionId": resp.json()["sessionId"]},
        )

if __name__ == "__main__":
    asyncio.run(main())

uv run main.py
Congratulations! You have successfully integrated Browser Cash with Crawl4AI. You can now leverage the power of Browser Cash’s stealth browsers to crawl and extract data from even the most challenging websites.