ai-job-search

The duplicate occurs because tools/job_key.py hashes title + url; non-Latin titles slugify to untitled, while Freehire's alphanumeric slugs do not match the six-digit _JOB_ID pattern. Fix the helper by hashing the URL alone whenever a URL is present, and use the title only as a fallback when no URL exists. Then verify that all title variants produce one key:

U=https://freehire.me/jobs/inzhener-ooo-chen-hlk3qjfg
python3 tools/job_key.py --company "ООО Чен" --title "Инженер" --url "$U"
python3 tools/job_key.py --company "ООО Чен" --title "инженер" --url "$U"
python3 tools/job_key.py --company "ООО Чен" --title "Инженер-механик" --url "$U"
python3 tools/job_key.py --audit

All three commands should return the same key, and duplicate_urls should no longer report the posting. Rebuild or deduplicate seen_jobs.json after applying the fix so old duplicate keys do not remain in the exclusion set. Do not rely on title slug normalization or alphanumeric slug IDs because the slug can change when a posting is re-listed.

This happens because convert_salary_excel.py does not pair bare headers such as Count + Index (or Danish Antal + Lønindeks). Their derived category names are empty, so they are exported as separate categories instead of one all_employees category.

Immediate workaround: give both headers the same non-empty category suffix, then regenerate the JSON:

Count all | Index all

or in Danish:

Antal alle | Lønindeks alle

Then run:

python3 tools/convert_salary_excel.py acme.xlsx --output salary_data.json
python3 salary_lookup.py "Acme Corp"

The lookup should then show one combined row with the count, index, and baseline variance.

Permanent code fix: change the count/index pairing condition in tools/convert_salary_excel.py to accept equal derived category names including the empty string, and assign an empty pair the default category name all_employees. A bare Count/Antal column without a matching index column must remain a standalone count column.

The detail command calls fetch() directly in .agents/skills/jobdanmark-search/cli/src/commands/detail.ts, bypassing the CLI's retry wrappers. Fix it by adding an htmlFetch(url) wrapper in jobdanmark-search/cli/src/helpers.ts that uses the same retry/backoff loop, timeout, and exported USER_AGENT as apiFetch/apiPost. Have it return null for HTTP 404 so detail preserves its NOT_FOUND behavior, then replace the direct fetch() call in detail.ts with htmlFetch(). Extend the retry-backoff, request-timeout, and wrapper tests, and add a handler-level test that stubs fetch to return 429 and verifies multiple attempts with the expected 500ms-to-5s backoff schedule.

The crash happens because Python's piped stdout on Windows defaults to the legacy ANSI code page (cp1252), which cannot encode non-Latin characters (e.g. Polish Ż, Ł; Danish Ø; Cyrillic/CJK). Fix it by forcing UTF-8 output at script startup. In tools/fetch_page.py and tools/robots_check.py, call the shared _force_utf8_output() helper before any print() — the same pattern already used in tools/rank_state.py, tools/job_key.py, tools/posting_guard.py, tools/verify_layout.py, tools/verify_pdf.py, and tools/upstream_triage.py. It reconfigures sys.stdout/sys.stderr to UTF-8 (equivalent to sys.stdout.reconfigure(encoding='utf-8', errors='replace') on Python 3.7+). Alternative workarounds without code changes: set environment variable PYTHONIOENCODING=utf-8 or PYTHONUTF8=1 before running, e.g. PYTHONIOENCODING=utf-8 python3 tools/fetch_page.py <url> --text, or run chcp 65001 in the Windows console to switch it to UTF-8.

In the JSON-LD branch (fromJsonLd in detail.ts), do not assign jobPosting.description verbatim. Parse it with the existing node-html-parser dependency and use structuredText to decode entities and convert block elements into separate lines, then normalize each line with the existing cleanText helper. This makes the JSON-LD output match the rendered-HTML branch: tags are stripped, entities such as & and ø are decoded, and paragraph/list line breaks are preserved. Update detail-jsonld.test.ts so it expects plain text instead of raw <p> markup, and add a test case containing HTML such as <p>...</p><ul><li>...</li></ul>.