PageIndex

Replace unsafe key deletion with pop() and a default value. In pageindex/page_index.py (or pageindex/page_index_classic.py), change:

del item_copy['page']
del item['page']

to:

item_copy.pop('page', None)
item.pop('page', None)

This removes the key when present and returns None instead of raising KeyError when a TOC item has no page field. Upgrading to the latest release may also resolve the issue because the legacy TOC path was replaced with additional validation, cleanup, retry, and error handling.

Guard the key removal instead of using del. In pageindex/page_index.py or pageindex/page_index_classic.py, replace:

del item['page']
del item_copy['page']

with:

item.pop('page', None)
item_copy.pop('page', None)

This safely removes page only when it exists. If running an installed release, upgrade to the latest version; the legacy TOC path was later replaced or covered by parser validation and error-handling changes. If the error persists, report the input shape, model/provider, package version, and traceback.

Fix the JSON fallback and add runtime type validation. In pageindex/utils.py, make extract_json() return [] instead of {} when parsing fails, because the ToC callers expect a list of dicts:

def extract_json(text):
    try:
        data = json.loads(text)
    except (json.JSONDecodeError, TypeError):
        return []

    return data if isinstance(data, list) else []

Normalize LLM output before using it:

def normalize_toc_items(data):
    if not isinstance(data, list):
        return []
    return [item for item in data if isinstance(item, dict)]

In meta_processor, filter only dict items before calling .get():

toc_with_page_number = normalize_toc_items(toc_with_page_number)
toc_with_page_number = [
    item for item in toc_with_page_number
    if item.get("physical_index") is not None
]

In process_no_toc, ensure both values are lists before calling .extend():

toc_with_page_number = normalize_toc_items(
    generate_toc_init(group_texts[0], model)
)

for group_text in group_texts[1:]:
    additional = normalize_toc_items(
        generate_toc_continue(toc_with_page_number, group_text, model)
    )
    toc_with_page_number.extend(additional)

Alternatively, update to a release containing the malformed-JSON crash guards from PRs #210, #217, or the comprehensive fix referenced in issue #199.

This was caused by an early-version bug where some parsed PDF entries lacked the physical_index field. Update PageIndex to the latest version, which includes the fix and regression test:

cd PageIndex
git pull
pip install -r requirements.txt

Then rerun your original PageIndex command. Python 3.9 was not confirmed as the cause. If the error or recognition problems persist with an old or complex PDF, PyPDF may fail to parse it reliably; as a workaround, use the hosted PageIndex API, which uses OCR and may handle such PDFs better.

Upgrade to the latest pageindex release, which adds client-side rate limiting, bounded concurrency, Retry-After handling, exponential backoff with jitter, and validation for malformed LLM JSON responses:

python -m pip install -U pageindex litellm

After upgrading, concurrent calls no longer burst through asyncio.gather, and an empty/failed LLM response is handled instead of accessing {}['toc_detected'].

If you cannot upgrade immediately, apply these workarounds:

1. Limit LLM concurrency with a semaphore:

sem = asyncio.Semaphore(2)  # match your provider's RPM/TPM tier

async def guarded_llm_call(coro):
    async with sem:
        return await coro

await asyncio.gather(*(guarded_llm_call(task) for task in tasks))

2. Replace the fixed 1-second retry with exponential backoff, jitter, and Retry-After support:

import asyncio, random

delay = min(60, (2 ** attempt) + random.random())
retry_after = getattr(exc, "response", None)
if retry_after is not None:
    delay = float(retry_after.headers.get("Retry-After", delay))
await asyncio.sleep(delay)

3. Validate parsed JSON before direct key access:

json_content = extract_json(response)
if not json_content or "toc_detected" not in json_content:
    raise RuntimeError("LLM returned an invalid or empty JSON response")
return json_content["toc_detected"]

Apply the same key-existence check to fields such as completed in the TOC completion checks.