mempalace

LoCoMo cuts the fused candidate list to top-k before llm_rerank_locomo() runs. The reranker therefore only reorders the same set of IDs, while compute_retrieval_recall scores evidence-set membership and ignores order. This is why runs with and without --llm-rerank produce identical recall, even though LLM calls are made.

Workaround: omit --llm-rerank for LoCoMo recall-only runs, for example:

python benchmarks/locomo_bench.py --granularity session --top-k 10 --mode hybrid

The flag can affect R@k on LongMemEval because reranking there occurs before the cut. To make it affect LoCoMo recall, rerank a wider candidate pool and select the final top-k items before truncation. A naive “pick k from the pool” prompt may reduce multi-evidence recall, so consider keeping the fused head and letting the LLM fill only tail slots. CLI help and benchmarks/BENCHMARKS.md were updated to document this limitation.

In versions 3.8.0–3.10.0 and develop, create_tunnel() in mempalace/palace_graph.py loads, scans, and rewrites the entire tunnels.json file for each tunnel. Mine loops call it repeatedly, causing quadratic time and excessive disk writes.

Fix it by batching tunnel creation:

1. Add a thread-local tunnel_batch() context manager that acquires the mine lock, loads tunnels.json once, and builds an id -> tunnel index.
2. Make create_tunnel() detect an active batch for the same file and insert/update through the index in O(1), without saving after each call.
3. Save the full tunnel list once when the batch context exits, preferably in finally.
4. Wrap entity_tunnels_for_wing(), compute_topic_tunnels(), and the loop in topic_tunnels_for_wing() with with tunnel_batch(config):.
5. Keep the existing load-mutate-save path outside the batch so MCP tools and explicit user tunnels remain unchanged.

Reference implementation:

_TUNNEL_BATCH = threading.local()

@contextmanager
def tunnel_batch(config=None):
    config = config or MempalaceConfig()
    tunnel_file = _get_tunnel_file(config)
    outer = getattr(_TUNNEL_BATCH, "state", None)
    if outer is not None and outer["file"] == tunnel_file:
        yield outer
        return
    with mine_lock(tunnel_file):
        tunnels = _load_tunnels(config)
        state = {
            "file": tunnel_file,
            "tunnels": tunnels,
            "index": {t.get("id"): t for t in tunnels if isinstance(t, dict)},
        }
        _TUNNEL_BATCH.state = state
        try:
            yield state
        finally:
            _TUNNEL_BATCH.state = outer
            _save_tunnels(state["tunnels"], config)

With this change, a refresh that previously performed about 8.8k full load/save cycles becomes one load and one save; the reported example dropped to about 8 seconds, dominated by the single JSON dump.

The bug is in ChromaCollection._lexical_search_via_sqlite in backends/chroma.py. The unfiltered SQLite FTS5 query applies LIMIT without ORDER BY, so SQLite returns matches in rowid/filing order. The first 500 matches are then BM25-scored, and a better match filed later is never considered.

Fix the capped query by ranking FTS candidates before applying the limit:

SELECT e.id, e.embedding_id
FROM embedding_fulltext_search
JOIN embeddings e ON e.id = embedding_fulltext_search.rowid
WHERE embedding_fulltext_search MATCH ? AND c.name = ?
ORDER BY bm25(embedding_fulltext_search)
LIMIT ?;

For SQLite FTS5, lower bm25() values are better. This matches the FTS rank ordering already used elsewhere. Then score the returned window and keep the top n_results.

Immediate workarounds on affected versions such as v3.10.0:

1. Upgrade to a release containing the fix from PR fix(search): rank the unfiltered lexical window before capping it.
2. If patching locally, add the ORDER BY bm25(...) above before LIMIT.
3. As a temporary workaround, pass a where filter; that code path already removes the 500-row cap.
4. Alternatively, remove or increase max_candidates=500, though this can increase query cost and only helps if the new cap covers all matches.

Increasing max_distance does not help because the missing drawer never enters either the lexical or vector candidate set.

The legacy mempalace repair path in mempalace 3.9.0/3.10.0 does not hold mine_palace_lock during _extract_drawers, check_extraction_safety, and copy_palace_dir. A hook-launched mempalace mine process can acquire the lock during that window, causing repair to fail later with MineAlreadyRunning and discard the extracted/backup data.

Permanent fix: install a release containing the fix for legacy repair, or patch cli/cmd_repair.py so the entire operation from backend.get_collection through _post_rebuild_cleanup runs inside:

with mine_palace_lock(palace_path):
    ...

It should catch MineAlreadyRunning and exit non-zero with the lock holder before extraction starts. The per-batch lock acquisitions in the Chroma backend are re-entrant on the same thread.

Immediate workaround on affected versions: disable the Claude Code hook that starts miners, ensure no miner is running, then run repair:

pkill -f 'mempalace mine' || true
mempalace repair --yes

If the hook cannot be disabled, prevent miners for the whole repair:

while true; do
  pkill -f 'mempalace mine' || true
  sleep 2
done &
killer=$!

mempalace repair --yes
rc=$?

kill "$killer"
exit "$rc"

A remaining locks/mine_palace_<hash>.lock file is normally inert because the flock is released when the process exits. Remove it only after confirming no repair or miner process is still running and new miners are still refused.

Upgrade MemPalace to a release containing PR #2593. This affects MemPalace 3.9.0: the MCP server's cached ChromaDB client did not update its chroma.sqlite3 freshness timestamp after the server's own writes, so mempalace_add_drawer, mempalace_check_duplicate, and mempalace_diary_write incorrectly treated their own writes as external changes and reloaded the HNSW index. Update with pip install --upgrade mempalace, then confirm the installed version includes #2593 with pip show mempalace. If no fixed release is available yet, install the patched build from PR #2593. Search calls are unaffected because the ChromaBackend path already received the own-write re-stamp in #2307; after the patch, writes from other processes are still detected on the next change.