Question 1

How to fix LocalAI ignoring model-specific `context_size` and always using the global Default Context Size?

Accepted Answer

In LocalAI v4.11.0, place `context_size` at the top level of the model YAML, not under `parameters`. The loader ignores `parameters.context_size`, so the global Default Context Size is used instead.

Correct configuration:

```yaml
backend: llama-cpp
context_size: 8192

parameters:
  model: Qwen3-VL-4B-Thinking-Q4_K_M.gguf
```

After saving the change, restart LocalAI and verify the startup log:

```text
load_model: initializing, n_slots = 1, n_ctx_slot = 8192
```

The top-level `context_size` takes precedence over the global Default Context Size. If the YAML was generated by an affected gallery template, move `context_size` out of the `parameters` block manually; a gallery fix places this option at the YAML root.

Question 2

How to fix 'reasoning_effort=none' not working in LocalAI 4.3.4?

Accepted Answer

This is a known regression in LocalAI versions after 4.0.0 (issue #10072). The parameter `reasoning_effort=none` should prevent the model from producing reasoning tokens and speed up responses, but a bug in newer versions causes it to be ignored. As a temporary workaround, downgrade to LocalAI v4.0.0 or v3.12.1, where the feature was reported to function correctly with llama-cpp backend models like Qwen3. If downgrading is not possible, you can also try forcing the model to skip reasoning by setting `top_p=0` and `temperature=0`, or using a non-reasoning model for latency-sensitive tasks. For a permanent fix, monitor the GitHub issue #10072 and upgrade once the patch is released. Ensure your model configuration file correctly maps the `reasoning_effort` option to the backend parameter (e.g., in llama-cpp, it should map to `--reasoning-effort none`).

Question 3

Why are some LocalAI v4.3.2 Docker images missing from Docker Hub?

Accepted Answer

A CI build failure prevented publishing of several v4.3.2 tags. Affected missing tags: v4.3.2, v4.3.2-gpu-nvidia-cuda-12, v4.3.2-gpu-nvidia-cuda-13, v4.3.2-gpu-vulkan, v4.3.2-gpu-intel. Successfully published tags: v4.3.2-gpu-hipblas, v4.3.2-nvidia-l4t-arm64, v4.3.2-nvidia-l4t-arm64-cuda-13. As a workaround, use the `localai/localai:master` image.

Question 4

Why does LocalAI return reasoning/thinking output inside the content field instead of a separate reasoning field?

Accepted Answer

This is a known regression in LocalAI v4.3.1 that affects gallery-provided models (e.g. qwen3-4b) whose configs predate the model import code: the `...` reasoning is merged into `content` instead of being split into the `reasoning` field. It is fixed by PR #9991 — upgrade your LocalAI container image to a release that includes that fix (e.g. pull the latest tag instead of `localai/localai:v4.3.1-gpu-vulkan`). As a workaround, downgrade to v4.0.0 where splitting works correctly, or use an imported (non-gallery) model, since gallery models installed before the import code landed are the ones affected. Reinstalling backends/deleting the data directory does NOT resolve it on v4.3.1.

Question 5

How do I fix duplicate `tool_calls` with different `index` values in LocalAI streaming `/v1/chat/completions`?

Accepted Answer

This occurs in LocalAI's streaming path when the C++ chat-template autoparser and the Go iterative JSON parser both emit the same tool call. Each parser uses its own index tracking, so one logical call can appear at indices `0`, `1`, and sometimes `2`. Non-streaming requests are unaffected.

Immediate workaround: disable streaming for tool-calling requests:

```json
{
  "stream": false
}
```

For example:

```bash
curl -s -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "orca-agent-v0.1",
    "messages": [{"role":"user","content":"Run ls /tmp | wc -l"}],
    "tools": [{
      "type":"function",
      "function":{
        "name":"bash",
        "description":"Run a bash script",
        "parameters":{
          "type":"object",
          "properties":{"script":{"type":"string"}},
          "required":["script"]
        }
      }
    }],
    "stream": false
  }'
```

The following do not resolve the duplication: `disable_peg_parser: true`, `grammar.no_mixed_free_string: true`, or `parallel_tool_calls: false`; the latter is not honored by `/v1/chat/completions`.

Code-level fix: in `core/http/endpoints/openai/chat.go`, use only one tool-call parser per request. If tool calls are already present in `chatDeltas` from the C++ autoparser, skip `functions.ParseJSONIterative`; alternatively, initialize and advance `lastEmittedCount` using the C++-emitted calls and deduplicate the deferred flush in `chat_emit.go`.

LocalAI

Core Features

What It Can't Do

Use Cases

Tags

Getting Started

Similar Projects