iFixAi
CLI Toolifixai-ai/iFixAi
Independent auditing tool that verifies AI agents actually do their intended job.
Overview
iFixAi is an independent auditing tool for AI agents that answers whether an agent is doing the job it is supposed to do, based on business KPIs rather than just technical metrics. It blends adversarial red-teaming depth with operational assurance discipline, runnable via a guided CLI wizard, scriptable flags, or agent plugins. It returns JSON and Markdown scorecard reports in under two minutes.
README Preview
\n \n \n \n \n \n \n\n\n\n English · 简体中文 · 日本語 · 한국어\n\n\nCatch your agent's mistakes and blind spots before the shit hits the fan.\n\n\n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n\n\n\n \n \n\n\n\n Quick start •\n Three ways to run •\n Test your agent •\n Scoring •\n Audit recorded actions •\n Docs •\n Contributing\n\n\n\n \n \n \n \n \n \n \n\n\n---\n\n## What it is\n\nThe existing Eval, Red-teaming, and Observability Tools are evaluating the agent mainly based on tech capability (token efficiency, latency, prompt injections). They cannot answer the most crucial question.\n\nIs the agent doing the job it is supposed to do based on the business KPIs and Organizational Structure? iFixAi gives you this answer in less than 120 seconds by striking the right balance between AI-Red Teaming and Operational Assurance. \n\nAdversarial depth. Assurance discipline. All-in-one auditing process.\n\n## Three ways to run\n\nAll three run the same diagnostic underneath. The difference is how you configure and drive it.\n\n| | **CLI: guided wizard** | **CLI: explicit flags** | **Plugin or Skill** |\n|---|---|---|---|\n| **How you drive it** | `ifixai setup` once → `ifixai run` zero-flag every time; config saved to `ifixai.yaml` | pass every option as a CLI flag; fully scriptable | the agent is the operator: discovers your setup, builds the fixture, runs it, and explains the scorecard |\n| **Best for** | first-time users, fast repeatable runs, team onboarding | CI, automation, audit-ready scripted batches | a guided, explained run with an interactive scorecard, inside the agent you already use |\n| **Setup** | `pip install "ifixai[]"` + `ifixai setup` | `pip install "ifixai[]"` + export keys | Claude Code or Codex: install the plugin (self-provisions). Any agent: `uvx ifixai install` scaffolds `/ifixai-skill` |\n| **Keys** | auto-detected by wizard; stored as env-var name in `ifixai.yaml`, never the secret itself | `--ap
FAQ (1)
TroubleshootingWhy does ifixai judge connect to localhost:8000 instead of my custom endpoint in self-eval mode, and how to fix ProviderConnectionError?
Root cause: ifixai/providers/http.py line 22 hardcodes DEFAULT_ENDPOINT = "http://localhost:8000/v1", and the --endpoint CLI value is not propagated through JudgeConfig, so the judge's internal HTTP provider falls back to the hardcoded default while the SUT uses your custom endpoint. Workaround: patch line 22 to read an environment variable: DEFAULT_ENDPOINT = os.environ.get("IFIXAI_HTTP_ENDPOINT", "http://localhost:8000/v1"), then run export IFIXAI_HTTP_ENDPOINT="https://my-proxy.example.com/v1" before ifixai run. Alternatively, temporarily edit DEFAULT_ENDPOINT directly to your proxy URL. Permanent fix requires propagating the endpoint through JudgeConfig (tracked in issues #14 and #15).