Question 1

How to fix 'ValueError: Cannot index mlx array using the given type' when running llama model with airllm on Mac?

Accepted Answer

This error occurs when using airllm with the MLX backend on Apple Silicon because the model expects MLX arrays, but the input token IDs are provided as a PyTorch tensor. To fix, convert the input tokens: `import mlx.core as mx; generation_output = model.generate(mx.array(input_tokens['input_ids']))` instead of `input_tokens['input_ids'].cuda()`. This ensures compatibility with MLX's embedding layer. The issue was reported with airllm v2.9.1 on Python 3.12.4, mlx versions 14.1–16.1, and macOS 14.5 on M1 Pro.

Question 2

Does AirLLM natively support Llama3 70B?

Accepted Answer

AirLLM already supports Llama3 70B natively, as Llama3 has no architecture changes that require adjustments to AirLLM's existing Llama implementation. No additional feature development is needed to run Llama3 70B with AirLLM.

Question 3

How to fix "ModuleNotFoundError: Could not import module 'LlamaForCausalLM'" when loading a model with transformers/airllm on macOS?

Accepted Answer

This error is misleading — it's a masked import failure caused by a broken `torchvision` installation, not a missing Llama module. The real error in the traceback is `RuntimeError: operator torchvision::nms does not exist`, which happens when `torch` and `torchvision` versions are binary-incompatible (common on Python 3.13 / Apple Silicon). Transformers' lazy loader triggers `image_utils` → `torchvision`, and the crash surfaces as the confusing ModuleNotFoundError.

Fix for text-only LLMs (Qwen, Llama, etc.) — remove torchvision entirely:
```bash
uv pip uninstall torchvision
# or: pip uninstall -y torchvision
```
Without torchvision, transformers skips the vision import path and `LlamaForCausalLM` loads normally.

If you do need vision support, reinstall matched versions instead:
```bash
uv pip install -U --force-reinstall torch torchvision
```
Ensure both packages come from the same release channel (e.g., torch 2.5.x with torchvision 0.20.x). Verify with `python -c "import torchvision; print(torchvision.__version__)"`.

Note for macOS users: also replace `.cuda()` with `.to('mps')` or run on CPU, since CUDA is unavailable on Apple Silicon.

Question 4

How to fix AirLLM 'Cannot index mlx array using the given type' error on macOS Apple Silicon?

Accepted Answer

AirLLM's MLX backend (`AirLLMLlamaMlx.generate()`) only accepts `mlx.array` indices, but the HuggingFace tokenizer with `return_tensors="pt"` returns a `torch.LongTensor`, which crashes `mlx.nn.Embedding`. This affects all architectures on macOS (Llama, Qwen, Mistral, Mixtral, GLM, Baichuan) because `AutoModel` routes everything through the MLX class. Workaround — coerce the token IDs to an mlx array before calling generate():

```python
import mlx.core as mx
from airllm import AutoModel

model = AutoModel.from_pretrained("Qwen/Qwen2.5-7B-Instruct", compression=None)
tokens = model.tokenizer(["Hello world"], return_tensors="pt").input_ids
input_ids = mx.array(tokens.detach().cpu().numpy())  # torch -> numpy -> mlx
model.generate(input_ids, max_new_tokens=8)
```

Alternatively tokenize with `return_tensors="np"` and wrap with `mx.array(...)`. A permanent fix (an internal `_coerce_to_mlx_array()` helper handling torch/numpy/mlx inputs at the `generate()` boundary) has been proposed via PR; until merged, manual conversion is required.

airllm

Overview

README Preview

FAQ (4)

同类型项目

browser-use

puppeteer

manim

PaddleOCR