sie
Applicationsuperlinked/sie
Self-hosted inference engine serving many open models for AI agents via one API.
Overview
SIE is an open-source inference server and production cluster that runs search, document-to-markdown, structured output, content safety, and agent-loop models behind one OpenAI-compatible API. It loads models on demand, supports 100+ pre-configured models, and ships Kubernetes deployment configs for production use.
README Preview
\n\n\n \n \n\n\nSIE: Superlinked Inference Engine\n\nSelf-hosted inference for agents. Every open model your agents call, served from one cluster in your cloud.\n\n\n Docs |\n Quickstart |\n API Reference |\n Models\n\n\n[](LICENSE)\n[](https://pypi.org/project/sie-sdk/)\n[](https://github.com/superlinked/sie/stargazers)\n\n⭐ _Help us reach more developers and grow the SIE community. Star this repo!_\n\n\n\n## About\n\nSIE is an open-source inference engine that runs the models behind every agent task through one API: search and retrieval, document-to-markdown conversion, structured output, content safety, and the agent loop itself. It replaces the patchwork of a separate model server per task with one system that serves 100+ models, loading each on demand.\n\n- OpenAI-compatible API for drop-in migration: `/v1/embeddings`, `/v1/chat/completions`, `/v1/completions`, `/v1/responses`\n- Pre-configured model catalog: Stella, SPLADE, Qwen3, GLiNER, SigLIP, and more; embedding and retrieval models benchmarked on MTEB\n- Serves multiple models simultaneously with on-demand loading and LRU eviction\n- Ships Kubernetes and Helm deployment configs for the load-balancing gateway, KEDA autoscaling, and Grafana dashboards\n- Integrates with LangChain, LlamaIndex, Haystack, DSPy, CrewAI, Chroma, Qdrant, Weaviate, and LanceDB\n\n## Development\n\nInstall [mise](https://mise.jdx.dev/getting-started.html), then bootstrap the\nversioned Python, Rust, Node.js, and Helm toolchains from the repository root:\n\n```bash\n./tools/init.sh\n```\n\nThe common development checks and local server are available as mise tasks:\n\n```bash\nmise run test\nmise run lint\nmise run typecheck\nmise run serve\nmise run rust-check\nmise run rust-test\nmise run gateway-test\nmise run server-sidecar-test\n```\n\nThe Python workspace uses the committed root lock. Package membership is\nexplicit in `pyproject.toml`; a package joins the workspace only in the same\nchange that adds its complete source.
FAQ (5)
TroubleshootingHow to fix Whisper HTTP 500 for concurrent transcriptions of audio longer than 30 seconds with different lengths?
This occurs because sie-server batches long-form Whisper requests, but Transformers' pipeline collate cannot handle log-mel tensors with different frame counts, such as [128, 18110] and [128, 18593]. It affects sie-server 0.9.0 with transformers 4.57.6 when two recordings longer than 30 seconds are processed in the same batch.
Immediate workaround: set Whisper pipeline_batch_size to 1 in the server configuration and restart the server. This processes each recording separately and avoids the failing collate path. Clips of 30 seconds or shorter are unaffected.
Permanent fix: in packages/sie_server/src/sie_server/adapters/whisper/adapter.py, keep batch_size=self._pipeline_batch_size for the short-form group, but run the long-form group with batch_size=1, one recording per pipeline call. For direct Transformers usage, call the pipeline separately for each long-form audio item instead of passing multiple unequal-length items in one batch.
After the fix, two recordings longer than 30 seconds with different durations should return transcripts in one extract() call, and concurrency 4/8 should complete without HTTP 500. Measured long-form throughput remains effectively unchanged: about 21.9 audio-seconds/second at concurrency 4 with batch_size=1 versus 22.3 at concurrency 1 with the default configuration.
TroubleshootingHow do I fix sie_mcp OAuth issuer and token_endpoint being spoofed through X-Forwarded-Host?
Upgrade to a sie_mcp release containing PR #292 (commit f5c4451). As an immediate workaround, explicitly pin the public origin so X-Forwarded-Host and X-Forwarded-Proto are ignored:
export SIE_MCP_PUBLIC_URL=https://mcp.your-domain.example
For Helm, set:
mcpEdge:
publicUrl: "https://mcp.your-domain.example"
oauthEnabled: true
After the fix, if SIE_MCP_PUBLIC_URL is unset, request-origin derivation is allowed only for loopback hosts or hosts listed in SIE_MCP_ALLOWED_HOSTS; untrusted hosts receive HTTP 503 from OAuth metadata routes and no WWW-Authenticate challenge. The updated Helm chart also renders SIE_MCP_PUBLIC_URL=https://<mcpEdge.ingress.host> automatically when mcpEdge.publicUrl is empty and edge ingress is enabled. Ensure your reverse proxy or load balancer overwrites/sanitizes X-Forwarded-Host, and do not enable ingress-nginx use-forwarded-headers unless those headers are trusted.
TroubleshootingWhy do Docker image builds install different dependency versions than the ones pinned in uv.lock?
This happened because the image builders ran resolve-deps to write each bundle's ranged dependency specifiers (e.g. gliner>=0.2.26,<1) to a requirements file, then ran pip install --target=/app/bundle-libs -r, which resolved ranges to the newest PyPI release at build time, shadowing the versions pinned in uv.lock that CI tests against. Two builds of the same commit could therefore ship different versions. Fixed by PRs #653 and #802: bundle images no longer resolve ranges during the build. Instead they install a saved dependency closure from checked-in per-target lock files. The 15 checked-in target locks cover all 11 released Linux amd64 images plus the four supported CPU arm64 combinations. Packages keep their root uv.lock pins unless an explicit, documented bundle exception applies (e.g. the transformers 5 bundle). CI rejects stale or inconsistent locks, and image builds verify and retain the installed /app/bundle-libs inventory (resolved pip freeze) so any deployed version can be audited. Upgrading to a release containing these fixes makes builds reproducible: building the same commit twice now installs identical versions.
TroubleshootingHow do I fix the keda-apply Helm hook being OOMKilled with exit 137 during upgrades?
This occurs in chart 0.7.2 because the keda-apply hook has a hardcoded 256Mi memory limit. Upgrades fetch existing ScaledObject resources as JSON and process them with jq, using more memory than a fresh install. Upgrade to a chart version containing PR #268, which adds configurable hook resources and raises the default limit to 1Gi. Then set, for example:
hooks:
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 200m
memory: 1Gi
Apply the values with helm upgrade <release> <chart> -f values.yaml. The configurable resources apply to the keda-apply, keda-cleanup, and KEDA ready hook Jobs. As a temporary workaround on older charts, you can set upgrade.disableHooks: true, but this is not recommended because it skips ScaledObject reconciliation and health checks. Helm post-renderers cannot be used to patch hook Jobs.
TroubleshootingHow do I fix incorrect or unlabelled prompt_tokens in the single-node /v1/embeddings response?
Upgrade to a release containing the fix fix(server): report measured embedding token usage (the fix for issue #566). On the single-node OpenAI-compatible route, the server now passes pipeline timing/usage through so that usage.prompt_tokens and usage.total_tokens use the worker's exact post-tokenization count when available. If no measured count exists, it falls back to the previous character estimate (len(input) / 4, minimum 1) because OpenAI clients expect a usage object. The response also includes usage.sie_token_source, set to worker for measured counts or character_estimate for the fallback. As a workaround before upgrading, use the native POST /v1/encode/{model} endpoint for exact counts, or place the service behind the gateway, which already reports sie_token_source as worker or character_estimate.