Scrapling
FrameworkD4Vinci/Scrapling
Adaptive web scraping framework with anti-bot bypass and auto-relocation.
Overview
Scrapling is an adaptive web scraping framework that handles everything from a single request to full-scale crawling. Its parser learns from website changes and automatically relocates elements when pages update. It bypasses anti-bot systems like Cloudflare Turnstile out of the box and supports concurrent multi-session crawls with proxy rotation.
README Preview
\n\n\n \n \n \n \n \n \n \n Effortless Web Scraping for the Modern Web\n\n\n\n \n \n العربيه | Español | Português (Brasil) | Français | Deutsch | 简体中文 | 日本語 | Русский | 한국어\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n Selection methods\n ·\n Fetchers\n ·\n Spiders\n ·\n Proxy Rotation\n ·\n CLI\n ·\n MCP\n\n\nScrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl.\n\nIts parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation - all in a few lines of Python. One library, zero compromises.\n\nBlazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.\n\n```python\nfrom scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher\nStealthyFetcher.adaptive = True\np = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # Fetch website under the radar!\nproducts = p.css('.product', auto_save=True) # Scrape data that survives website design changes!\nproducts = p.css('.product', adaptive=True) # Later, if the website structure changes, pass `adaptive=True` to find them!\n```\nOr scale up to full crawls\n```python\nfrom scrapling.spiders import Spider, Response\n\nclass MySpider(Spider):\n name = "demo"\n start_urls = ["https://example.com/"]\n\n async def parse(self, response: Response):\n for item in response.css('.product'):\nFAQ (5)
TroubleshootingHow to fix Docker MCP connection error: 'pull access denied for scrapling, repository does not exist' in Claude Desktop?
Use the full Docker image name pyd4vinci/scrapling instead of scrapling in claude_desktop_config.json.
1. Pull the image:
docker pull pyd4vinci/scrapling2. Update the MCP server configuration:
{
"mcpServers": {
"ScraplingServer": {
"command": "/usr/local/bin/docker",
"args": ["run", "-i", "--rm", "pyd4vinci/scrapling", "mcp"]
}
}
}On macOS, command can also be "docker" or /usr/local/bin/docker depending on your Docker installation.
3. Save the file and fully restart Claude Desktop.
The error occurs because Docker interprets scrapling as an image named scrapling:latest, but the actual pulled image is pyd4vinci/scrapling:latest.
TroubleshootingHow to fix "DeprecationWarning: The 'strip_cdata' option of HTMLParser()" in scrapling with lxml 6?
This warning appears because scrapling (≤0.4.15) passes strip_cdata to lxml's HTMLParser(), which lxml 6 deprecated (the option never had any effect and will be removed, after which it will raise TypeError). Fixes:
1. **Upgrade scrapling** — the maintainer removed the option in the dev branch, so update to the release after 0.4.15: pip install -U scrapling
2. **Temporary workaround — pin lxml below 6**: pip install "lxml<6"
3. **Suppress the warning** if you must stay on current versions: add warnings.filterwarnings("ignore", message=".*strip_cdata.*", category=DeprecationWarning) before creating the Selector.
TroubleshootingWhy does Scrapling StealthyFetcher fetch() take much longer than the timeout when solve_cloudflare=True?
In Scrapling 0.4.15, the Cloudflare Turnstile solver waits on a widget locator that never matches some challenge variants, and its retry loop is unbounded — so a fetch(..., timeout=15000) call can run 74s+ (the timeout argument is a per-operation limit, not a total fetch deadline). Fixes and workarounds:
1. **Upgrade**: Fixed on the dev branch for the next release — the wait for the missing widget was removed and the solver retry loops were capped. Install the dev build or upgrade past 0.4.15 when released.
2. **Reuse a session**: The cf_clearance cookie persists, so only the first fetch hits the solver. Measured: ~8s first page, then ~3s per page afterwards.
3. **External wall clock**: Run fetches in separate worker processes and kill any that overrun your deadline, since timeout= alone does not bound the call on affected versions.
Note the call still succeeds with status 200 and correct content — this is purely a latency bug, not a failure.
TroubleshootingHow to fix StealthyFetcher TargetClosedError when running as administrator on Windows?
When Python runs elevated (admin) on Windows, Scrapling's extra --disable-features switch replaces patchright/Playwright's list, dropping AutoDeElevate. Chromium then relaunches de-elevated and exits, causing TargetClosedError: BrowserType.launch_persistent_context. Workaround for scrapling 0.4.15: pass the de-elevation flag explicitly:
from scrapling.fetchers import StealthyFetcher
StealthyFetcher.fetch("https://example.com", headless=True, extra_flags=["--do-not-de-elevate"])Permanent fix: the disabled-features lists are merged in the dev branch, so upgrade to the next release after 0.4.15. Alternatively, run the script from a non-elevated shell.
TroubleshootingWhy is the ClawHub page for Scrapling skill blank or broken?
This was a temporary outage (openclaw/clawhub#2345) that has been resolved. The page at https://clawhub.ai/D4Vinci/scrapling-official is now working. If you still see an empty page, try using the zip file from the agent-skill GitHub directory as a fallback: https://github.com/D4Vinci/Scrapling/tree/main/agent-skill.