transformers

Known bug in transformers 5.x (5.18.0+, still on main): MMGroundingDinoForObjectDetection._tied_weights_keys ties every decoder layer's box head and class head to layer 0, ignoring the checkpoint config decoder_bbox_embed_share: false / decoder_cls_embed_share: false. Layers 1-5 therefore reuse layer 0's head and boxes come out wrong (e.g. RefCOCO acc@0.5 drops from 7/8 to 0/8). Affected checkpoints include iSEE-Laboratory/llmdet_tiny/base/large and openmmlab-community/mm_grounding_dino_tiny_* and mm_grounding_dino_large_all.

Fix option 1 — downgrade to the v4 behavior where each layer kept its own head:
pip install "transformers==4.57.1"

Fix option 2 — restore each decoder layer's own heads from the checkpoint after loading:

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

weights = load_file(hf_hub_download(model_id, "model.safetensors"))
for i in range(1, model.config.decoder_layers):
    model.bbox_embed[i] = type(model.bbox_embed[0])(model.config.d_model, model.config.d_model, 4, 3)
    model.bbox_embed[i].load_state_dict({k.split(f"bbox_embed.{i}.")[1]: v for k, v in weights.items()
                                         if k.startswith(f"model.decoder.bbox_embed.{i}.")})
    model.class_embed[i] = type(model.class_embed[0])(model.config)
    model.class_embed[i].bias.data = weights[f"model.decoder.class_embed.{i}.bias"].clone()
model.model.decoder.bbox_embed = model.bbox_embed
model.model.decoder.class_embed = model.class_embed

This is a bug in default_sample_indices_fn() in src/transformers/video_utils.py. It computes num_frames = int(total_num_frames / video_fps * fps), which truncates to 0 for short videos (e.g., a 10-frame, 30 FPS video sampled at 1 FPS gives int(10/30*1) == 0). Downstream decoders then crash: read_video_pyav() fails on indices[-1] and read_video_opencv() fails at np.stack(frames). Fix: upgrade transformers to a version containing the patch, which clamps the result so at least one frame index (e.g., [0]) is returned for any non-empty video with positive requested FPS. Workaround on older versions: request a higher fps so int(total_num_frames / video_fps * fps) >= 1, or monkey-patch the function with num_frames = max(1, int(total_num_frames / video_fps * fps)) when total_num_frames > 0 and fps > 0.

VLM processors in Hugging Face Transformers accept inconsistent input formats, so a pipeline call that works for one model may crash for another. Key rules and workarounds:

1. **URLs**: Only Idefics, Idefics2, Idefics3 and Pixtral accept image URLs directly. For all other models (LLaVA, BLIP, PaliGemma, Qwen2-VL, etc.), download and convert to PIL first:

from PIL import Image
import requests
image = Image.open(requests.get(url, stream=True).raw)

2. **File paths**: No VLM processor accepts raw file paths — always load with Image.open(path) first.

3. **Nested batches with a different number of images per text** (e.g. images=[[img1, img2], [img3]]): Only Qwen2-VL, Chameleon, PaliGemma, Llava-Next, LLaVA-OneVision, Mllama, InstructBLIP, Idefics2 and Idefics3 support this. Models like LLaVA, BLIP2, VipLlava and Pixtral will crash at the processor level. Workaround: switch to a compatible model (e.g. llava-hf/llava-onevision-qwen2-0.5b-ov-hf instead of llava-hf/llava-interleave-qwen-0.5b-hf), or process each text/image set in a separate non-batched call.

4. **Padding quirks**: Qwen2-VL crashes on non-nested batched images when padding=True; Chameleon requires padding to be set manually to True to work at all.

Check the model's processor documentation or test with a single PIL image first, since PIL is the only universally supported format.

SpanBERT's pretrained weights are on the Hugging Face Hub and load with the standard BERT architecture:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("SpanBERT/spanbert-base-cased")
model = AutoModel.from_pretrained("SpanBERT/spanbert-base-cased")

SpanBERT/spanbert-large-cased is also available. However, the coreference head from the original papers (the TensorFlow 1.14 mandarjoshi90/coref implementation) was never ported to Transformers, so there is no built-in SpanBERT coreference pipeline. Workarounds for coreference resolution today:

1. Use the community model biu-nlp/f-coref (fastcoref), which provides a ready-to-use coreference pipeline.
2. Use neuralcoref (spaCy-based) as an alternative.
3. Fine-tune a span-classification head on top of the loaded SpanBERT AutoModel yourself if you need to replicate the paper's approach.

Root cause: Gemma4AudioAttention applies the attention mask via masked_fill(attention_mask.logical_not(), ...), which assumes a bool mask. But create_bidirectional_mask returns an additive float mask under eager (0.0 = allowed, finfo.min = disallowed), so logical_not() inverts the semantics — allowed positions get masked out and, with a fully-allowed mask, attention degenerates to near-uniform averaging regardless of content. Immediate workaround: force sdpa attention, which uses a correct bool mask — load the model with attn_implementation="sdpa" or call model.set_attn_implementation("sdpa"). Permanent fix (patch transformers): in Gemma4AudioAttention, handle both mask conventions — bool mask → masked_fill(mask.logical_not(), invalid_value); float mask → apply additively with attn_weights = attn_weights + attention_mask after the blocked 5-D conversion. Also make _convert_4d_mask_to_blocked_5d dtype-aware: for float masks use F.pad(..., value=torch.finfo(dtype).min) instead of value=False (which pads 0.0, meaning *allowed* in the float convention). Verify the fix by comparing eager vs sdpa outputs with identical seeds — max abs diff should drop from ~0.12 to ~1e-6.