Question 1

Why do API hooks fail with 'ImportError: __import__ not found' in crawl4ai v8.0.0 Docker?

Accepted Answer

This is a known regression in v8.0.0 caused by an over‑restrictive builtins whitelist (PR #1712). The hook manager's internal compilation step runs `exec("import asyncio", namespace)` which requires the `__import__` built‑in, but it was removed from the allowed list. Any user‑provided hook, even a trivial one, triggers this error.

**Workarounds:**
- **Downgrade** to v7.x (e.g., `unclecode/crawl4ai:v7.0.0`)
- **If you control the container**, edit `hook_manager.py` and either:
  1. Add `'__import__'` back to the `allowed_builtins` list, or
  2. Replace the `exec("import asyncio")` / `exec("import json")` lines with direct assignments:
     ```python
     import asyncio, json
     namespace['asyncio'] = asyncio
     namespace['json'] = json
     ```

Track issue #1878 for the permanent fix.

Question 2

Why are sentences returned in random order when using NlpSentenceChunking in crawl4ai?

Accepted Answer

This is a known bug in `crawl4ai` where `NlpSentenceChunking.chunk()` uses `list(set(sens))`, which shuffles sentences because Python `set` is unordered. The fix has been merged into the `develop` branch and will be included in the next release.

**Workaround**: Apply the fix manually by editing `chunking_strategy.py` and changing:
```python
return list(set(sens))  # BUG
```
to:
```python
return sens
```

Or install directly from the fix branch:
```bash
pip install git+https://github.com/unclecode/crawl4ai.git@develop
```

The fix preserves document order and keeps intentional duplicate sentences.

Question 3

How to fix 'Deserialization of type LLMTableExtraction is not allowed' error in crawl4ai Docker Job Queue API?

Accepted Answer

This bug was caused by `LLMTableExtraction` missing from the deserialization allowlist in `async_configs.py`. It is fixed in crawl4ai v0.8.7 and later. To resolve, upgrade to the latest version:

```bash
docker pull ghcr.io/unclecode/crawl4ai:latest
```

If using pip, update with `pip install crawl4ai>=0.8.7`. Restart your Docker container to apply the fix.

Question 4

Why does crawl4ai ignore semaphore_count and cause 429 'Too Many Requests' errors?

Accepted Answer

It's a known bug where `DeepCrawlStrategy` creates a `MemoryAdaptiveDispatcher` that ignores the `semaphore_count` setting from your `CrawlerRunConfig`, defaulting to 20 concurrent tasks and triggering rate limits. The fix reads `semaphore_count` from the config and passes it as `max_session_permit`.

**Solution: Upgrade to version 0.8.7 or later:**

```bash
pip install --upgrade crawl4ai
```

**If you need a temporary workaround**, apply this change to your local installation (pre‑0.8.7):

Edit `crawl4ai/async_webcrawler.py` near line 1032:

```python
max_session_permit = max(1, int(getattr(primary_cfg, "semaphore_count", 5) or 5))
dispatcher = MemoryAdaptiveDispatcher(
    max_session_permit=max_session_permit,
    ...
)
```

Or install the patched branch:

```bash
pip install git+https://github.com/hafezparast/crawl4ai.git@fix/maysam-dispatcher-semaphore-count-1927
```

After the fix, `semaphore_count` in your config will properly control concurrency and prevent 429 errors.

Question 5

Why doesn't enable_stealth=True hide navigator.webdriver in crawl4ai 0.8.6?

Accepted Answer

In crawl4ai 0.8.6, the `StealthAdapter` incorrectly imports `stealth_async`/`stealth_sync` from `playwright-stealth` 2.x, causing stealth to be silently skipped. Upgrade to crawl4ai 0.8.7, which updates the adapter to use the `Stealth` class:

```bash
pip install --upgrade crawl4ai==0.8.7
```

If you cannot upgrade immediately, a temporary workaround is to downgrade `playwright-stealth` to the 1.x series:
```bash
pip install 'playwright-stealth<2'
```
(Note that this may conflict with other dependencies.)

crawl4ai

Overview

README Preview

FAQ (5)

同类型项目

puppeteer

PaddleOCR

prisma

daisyui